System and method for position tracking using edge computing

ABSTRACT

A tracking system includes a camera subsystem that includes cameras that capture vide of a space. Each camera is coupled with a camera client that determines local coordinates of people in the captured video. The camera clients generate frames that include color frames and depth frames labeled with an identifier number of the camera and their corresponding timestamps. The camera clients generate tracks that include metadata describing historical people detections, tracking identifications, timestamps, and the identifier number of the camera. The camera clients send the frames and tracks to cluster servers that maintain the frames and tracks such that they are retrievable using their corresponding labels. A camera server queries the cluster servers to receive the frames and tracks using their corresponding labels. The camera server determines the physical positions of people in the space based on the determined local coordinates.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/104,323 filed Nov. 25, 2020, by Jon Andrew Crain et al., entitled“SYSTEM AND METHOD FOR POSITION TRACKING USING EDGE COMPUTING,” which isa continuation-in-part of:

U.S. patent application Ser. No. 16/663,633 filed Oct. 25, 2019, bySailesh Bharathwaaj Krishnamurthy et al., and entitled “SCALABLEPOSITION TRACKING SYSTEM FOR TRACKING POSITION IN LARGE SPACES”, nowU.S. Pat. No. 10,885,642 issued Jan. 5, 2021;

U.S. patent application Ser. No. 16/663,415 filed Oct. 25, 2019, byCaleb Austin Boulio et al., and entitled “SENSOR ARRAY FOR SCALABLEPOSITION TRACKING SYSTEM”, now U.S. Pat. No. 10,878,585 issued Dec. 29,2020;

U.S. patent application Ser. No. 17/018,146 filed Sep. 11, 2020, byCaleb Austin Boulio et al., and entitled “SENSOR ARRAY FOR SCALABLEPOSITION TRACKING SYSTEM”, which is a divisional of U.S. patentapplication Ser. No. 16/663,415 filed Oct. 25, 2019, by Caleb AustinBoulio et al., and entitled “SENSOR ARRAY FOR SCALABLE POSITION TRACKINGSYSTEM”, now U.S. Pat. No. 10,878,585 issued Dec. 29, 2020;

U.S. patent application Ser. No. 16/991,947 filed Aug. 12, 2020, byDeepanjan Paul et al., entitled “CUSTOM RACK FOR SCALABLE POSITIONTRACKING SYSTEM”, which is a continuation of U.S. patent applicationSer. No. 16/663,669 filed Oct. 25, 2019, by Deepanjan Paul et al., andentitled “CUSTOM RACK FOR SCALABLE POSITION TRACKING SYSTEM”, now U.S.Pat. No. 10,783,762 issued Sep. 22, 2020;

U.S. patent application Ser. No. 16/941,787 filed Jul. 29, 2020, bySailesh Bharathwaaj Krishnamurthy et al., entitled “TRACKING POSITIONSUSING A SCALABLE POSITION TRACKING SYSTEM”, which is a continuation ofU.S. patent application Ser. No. 16/663,432 filed Oct. 25, 2019, bySailesh Bharathwaaj Krishnamurthy et al., and entitled “TRACKINGPOSITIONS USING A SCALABLE POSITION TRACKING SYSTEM”, now U.S. Pat. No.10,769,450 issued Sep. 8, 2020;

U.S. patent application Ser. No. 16/941,825 filed Jul. 29, 2020, bySailesh Bharathwaaj Krishnamurthy et al., entitled “TRACKING POSITIONSUSING A SCALABLE POSITION TRACKING SYSTEM”, which is a divisional ofU.S. patent application Ser. No. 16/663,432 filed Oct. 25, 2019, bySailesh Bharathwaaj Krishnamurthy et al., and entitled “TRACKINGPOSITIONS USING A SCALABLE POSITION TRACKING SYSTEM”, now U.S. Pat. No.10,769,450 issued Sep. 8, 2020;

U.S. patent application Ser. No. 16/663,710 filed Oct. 25, 2019, bySailesh Bharathwaaj Krishnamurthy et al., and entitled “TOPVIEW OBJECTTRACKING USING A SENSOR ARRAY”;

U.S. patent application Ser. No. 16/663,766 filed Oct. 25, 2019, bySailesh Bharathwaaj Krishnamurthy et al., and entitled “DETECTING SHELFINTERACTIONS USING A SENSOR ARRAY”;

U.S. patent application Ser. No. 16/663,451 filed Oct. 25, 2019, bySarath Vakacharla et al., and entitled “TOPVIEW ITEM TRACKING USING ASENSOR ARRAY” now U.S. Pat. No. 10,943,287 issued Mar. 9, 2021;

U.S. patent application Ser. No. 16/663,794 filed Oct. 25, 2019, byShahmeer Ali Mirza et al., and entitled “DETECTING AND IDENTIFYINGMISPLACED ITEMS USING A SENSOR ARRAY”;

U.S. patent application Ser. No. 16/663,822 filed Oct. 25, 2019, byShahmeer Ali Mirza et al., and entitled “SENSOR MAPPING TO A GLOBALCOORDINATE SYSTEM”;

U.S. patent application Ser. No. 16/941,415 filed Jul. 28, 2020, byShahmeer Ali Mirza et al., and entitled “SENSOR MAPPING TO A GLOBALCOORDINATE SYSTEM USING A MARKER GRID”, which is a continuation of U.S.patent application Ser. No. 16/794,057 filed Feb. 18, 2020, by ShahmeerAli Mirza et al., and entitled “SENSOR MAPPING TO A GLOBAL COORDINATESYSTEM USING A MARKER GRID”, now U.S. Pat. No. 10,769,451 issued Sep. 8,2020, which is a continuation of U.S. patent application Ser. No.16/663,472 filed Oct. 25, 2019, by Shahmeer Ali Mirza et al., andentitled “SENSOR MAPPING TO A GLOBAL COORDINATE SYSTEM USING A MARKERGRID”, now U.S. Pat. No. 10,614,318 issued Apr. 7, 2020;

U.S. patent application Ser. No. 16/663,856 filed Oct. 25, 2019, byShahmeer Ali Mirza et al., and entitled “SHELF POSITION CALIBRATION INAGLOBAL COORDINATE SYSTEM USING A SENSOR ARRAY”;

U.S. patent application Ser. No. 16/664,160 filed Oct. 25, 2019, byTrong Nghia Nguyen et al., and entitled “CONTOUR-BASED DETECTION OFCLOSELY SPACED OBJECTS”;

U.S. patent application Ser. No. 17/071,262 filed Oct. 15, 2020, byShahmeer Ali Mirza et al., and entitled “ACTION DETECTION DURING IMAGETRACKING”, which is a continuation of U.S. patent application Ser. No.16/857,990 filed Apr. 24, 2020, by Shahmeer Ali Mirza et al., andentitled “ACTION DETECTION DURING IMAGE TRACKING”, now U.S. Pat. No.10,853,663 issued Dec. 1, 2020, which is a continuation of U.S. patentapplication Ser. No. 16/793,998 filed Feb. 18, 2020, by Shahmeer AliMirza et al., and entitled “ACTION DETECTION DURING IMAGE TRACKING”, nowU.S. Pat. No. 10,685,237 issued Jun. 16, 2020, which is a continuationof U.S. patent application Ser. No. 16/663,500 filed Oct. 25, 2019, byShahmeer Ali Mirza et al., and entitled “ACTION DETECTION DURING IMAGETRACKING”, now U.S. Pat. No. 10,621,444 issued Apr. 14, 2020;

U.S. patent application Ser. No. 16/857,990 filed Apr. 24, 2020, byShahmeer Ali Mirza et al., and entitled “ACTION DETECTION DURING IMAGETRACKING”, now U.S. Pat. No. 10,853,663 issued Dec. 1, 2020, which is acontinuation of U.S. patent application Ser. No. 16/793,998 filed Feb.18, 2020, by Shahmeer Ali Mirza et al., and entitled “ACTION DETECTIONDURING IMAGE TRACKING”, now U.S. Pat. No. 10,685,237 issued Jun. 16,2020, which is a continuation of U.S. patent application Ser. No.16/663,500 filed Oct. 25, 2019, by Shahmeer Ali Mirza et al., andentitled “ACTION DETECTION DURING IMAGE TRACKING”, now U.S. Pat. No.10,621,444 issued Apr. 14, 2020;

U.S. patent application Ser. No. 16/664,219 filed Oct. 25, 2019, byShahmeer Ali Mirza et al., and entitled “OBJECT RE-IDENTIFICATION DURINGIMAGE TRACKING”;

U.S. patent application Ser. No. 16/664,269 filed Oct. 25, 2019, byMadan Mohan Chinnam et al., and entitled “VECTOR-BASED OBJECTRE-IDENTIFICATION DURING IMAGE TRACKING”;

U.S. patent application Ser. No. 16/664,332 filed Oct. 25, 2019, byShahmeer Ali Mirza et al., and entitled “IMAGE-BASED ACTION DETECTIONUSING CONTOUR DILATION”;

U.S. patent application Ser. No. 16/664,363 filed Oct. 25, 2019, byShahmeer Ali Mirza et al., and entitled “DETERMINING CANDIDATE OBJECTIDENTITIES DURING IMAGE TRACKING”;

U.S. patent application Ser. No. 16/664,391 filed Oct. 25, 2019, byShahmeer Ali Mirza et al., and entitled “OBJECT ASSIGNMENT DURING IMAGETRACKING”;

U.S. patent application Ser. No. 16/664,426 filed Oct. 25, 2019, bySailesh Bharathwaaj Krishnamurthy et al., and entitled “AUTO-EXCLUSIONZONE FOR CONTOUR-BASED OBJECT DETECTION”;

U.S. patent application Ser. No. 16/884,434 filed May 27, 2020, byShahmeer Ali Mirza et al., and entitled “MULTI-CAMERA IMAGE TRACKING ONA GLOBAL PLANE”, which is a continuation of U.S. patent application Ser.No. 16/663,533 filed Oct. 25, 2019, by Shahmeer Ali Mirza et al., andentitled “MULTI-CAMERA IMAGE TRACKING ON A GLOBAL PLANE”, now U.S. Pat.No. 10,789,720 issued Sep. 29, 2020;

U.S. patent application Ser. No. 16/663,901 filed Oct. 25, 2019, byShahmeer Ali Mirza et al., and entitled “IDENTIFYING NON-UNIFORM WEIGHTOBJECTS USING A SENSOR ARRAY”; and

U.S. patent application Ser. No. 16/663,948 filed Oct. 25, 2019, byShahmeer Ali Mirza et al., and entitled “SENSOR MAPPING TO A GLOBALCOORDINATE SYSTEM USING HOMOGRAPHY”, which are all incorporated hereinby reference.

TECHNICAL FIELD

This disclosure relates generally to a system and method for positiontracking using edge computing.

BACKGROUND

Position tracking systems are used to track the physical positions ofpeople and/or objects. Position tracking people and objects within aspace pose several technical challenges. Existing systems use varioustechniques for tracking people and objects within the space. Forexample, the existing systems use computationally intensive trackingtechniques for tracking people and objects. These computationallyintensive tracking techniques require a significant amount of time whichmeans that these techniques are not compatible with real-timeapplications such as video streams. As such, the existing systems lackthe ability to provide reliable and accurate tracking techniquescompatible with real-time applications.

SUMMARY OF THE DISCLOSURE

Position tracking systems are used to track the physical positions ofpeople and/or objects in a physical space (e.g., a store). These systemstypically use a sensor (e.g., a camera) to detect the presence of aperson and/or object and a computer to determine the physical positionof the person and/or object based on signals from the sensor. In a storesetting, other types of sensors can be installed to track the movementof inventory within the store. For example, weight sensors can beinstalled on racks and shelves to determine when items have been removedfrom those racks and shelves. By tracking both the positions of personsin a store and when items have been removed from shelves, it is possiblefor the computer to determine which user in the store removed the itemand to charge that user for the item without needing to ring up the itemat a register. In other words, the person can walk into the store, takeitems, and leave the store without stopping for the conventionalcheckout process.

For larger physical spaces (e.g., convenience stores and grocerystores), additional sensors can be installed throughout the space totrack the position of people and/or objects as they move about thespace. For example, additional cameras can be added to track positionsin the larger space and additional weight sensors can be added to trackadditional items and shelves. There is a limit, however, to the numberof sensors that can be added before the computing capabilities of thecomputer are reached. As a result, the computing power of the computerlimits the coverage area of the tracking system.

One way to scale these systems to handle larger spaces is to addadditional computers and to divide the sensors amongst these computerssuch that each computer processes signals from a subset of the sensors.However, dividing the sensors amongst multiple computers introducessynchronization issues. For example, the sensors may not communicatesignals to their respective computers at the same time orsimultaneously. As another example, sensors may have different latencieswith their respective computers and thus, it may take more time forsignals from one sensor to reach a computer than signals from anothersensor. As a result, the sensors and computers become desynchronizedwith each other and it becomes more difficult for the computers todetermine, in a cohesive way, the position of persons or objects in thespace and when items were removed.

One embodiment of a system disclosed in the present application providestechnical solutions to the technical problems discussed above byassociating a different camera client to each camera, and migrating thecamera clients closer to the cameras. With this method, edge computingcan be implemented by the camera clients, and frames from videoscaptured by the cameras can be processed when they are received by thecamera clients. The disclosed system provides several practicalapplications and technical advantages which include: 1) a process foredge processing of the frames of videos captured by the cameras as theyare received by the camera clients by migrating the camera clientscloser to the cameras which improves synchronizations of the frames; 2)a process for generating frames at the camera clients, which includesindexing color frames and depth frames associated with the frames withtheir corresponding labels, such as camera identifiers, timestamps,etc.; 3) a process for generating tracks at the camera clients, wherethe tracks include metadata detected in the depth frames, such astracking identifications, historical detections (e.g., bounding areas,contours, segmentation masks), etc.; 4) utilizing a cluster of serversconfigured to store and maintain the frames and tracks using one or moreof their corresponding labels; 5) enabling to query any frame using oneor more of its corresponding labels; and 6) enabling to query any trackusing one or more of its corresponding labels.

As such, the disclosed system may improve the process of objecttracking, and more specifically improve the process of object trackingin a large space. The disclosed system may be integrated into apractical application of reducing the computing costs of the trackingprocess by migrating the camera clients closer to the cameras. This, inturn, provides an additional practical application of increasing theaccuracy of the object tracking process. By implementing the edgeprocessing of the color frames and depth frames, the color frames anddepth frames can be compressed before sending over to the clusterservers which provides another practical application of saving bandwidthof transmissions of the frames and tracks.

Furthermore, since the color frames and depth frames are processed atthe edge camera clients and streamed over to the cluster servers, thecamera clients do not need to wait for a certain number of frames toarrive to start a tracking process. Similarly, the camera server doesnot need to wait for a certain number of frames to arrive to start awindowing process. As such, the disclosed system provides anotherpractical application of improving the speed of the tracking process.The camera server can routinely perform the windowing processes, andrequest to receive additional frames and tracks from that time window atany time for processing. In other words, the camera server can referenceback in time and request frames and tracks with a particular timestamp(in a particular time window) for processing. With this approach,offline tracking algorithm testing can be leveraged where frames andtracks associated with different scenarios (different timestamps) can beused as a training dataset to further improve the tracking process. Forexample, frames and tracks from different timestamps can be pulled fromthe cluster servers to input to different versions of the trackingalgorithm (and likewise to different tracking algorithms). As such,different versions of the tracking algorithm (and likewise differenttracking algorithms) can further be tested using the training dataset.Furthermore, online or live tracking algorithm testing can be leveragedwhere one or more tracking algorithms can simultaneously process thesame incoming frames and tracks. Thus, with this approach, the trackingprocess can be further optimized.

One embodiment of a system disclosed in the present application providesa technical solution to the technical problems discussed above byassigning a particular address number to each weight sensor andconfiguring each weight sensor to communicate its corresponding addressnumber along with a signal indicative of a weight of an item to itscorresponding weight board. With this method, the location of an eventcorresponding to an item being removed from or put on a weight sensor isidentified based on the address number of that weight sensor.

As such, the disclosed system provides several practical applicationsand technical advantages which include: 1) a process for assigning aparticular address number to each weight sensor, where a particularaddress number of a weight sensor indicates a location of the weightsensor within a store; 2) a process for configuring each weight sensorto communicate its corresponding address number along with a signalindicative of a weight of an item to its corresponding weight board; and3) a process for utilizing a bus wire for connecting weight boards(associated with the weight sensors) to a port on a circuit board.

As such, the disclosed system may improve the item position trackingprocess and facilitate detecting a location where an item has beenremoved from or put on a weight sensor based on an address number ofthat weight sensor. The disclosed system may be integrated into apractical application of leveraging edge computing and thus producingmore accurate weight information of items on the weight sensors becausesignals from the weight sensors are processed by the weight boards thatare placed close to the weight sensors. The disclosed system may beintegrated into an additional practical application of providing a morescalable solution for the item position tracking process by assigningeach weight sensor with a particular address number because wires fromthe weight boards can be aggregated into a bus wire that connects to aport on a circuit board. Thus, the number of weight sensors can beadjusted without running out of ports on a circuit board, therebyobviating a need to use multiple circuit boards if the number of weightsensors becomes more than the number of ports on a circuit board.Furthermore, since each weight sensor is assigned a particular addressnumber, and signals from the weight sensors are transmitted wirelessly,racks within the store can be moved around and re-arranged moreflexibly. Once the racks are positioned according to the desired layout,the address numbers of the weight sensors are used to determine the newpositions of the weight sensors.

Another way to further improve the item tracking system is byimplementing an adaptive item counting algorithm when one or more itemsare removed from (or put on) a weight sensor. In some cases, theenvironmental changes, e.g., temperature changes, can affect an itemcounting process. For example, due to environmental changes, a weightsensor may exhibit a different (or unexpected) sensitivity compared toits default sensitivity, thus, a weight change corresponding to aquantity of an item being removed from (or put on) the weight sensor maynot be the actual item quantity that was removed from (or put on) theweight sensor. In other words, the sensitivity of the weight sensor mayshift from the default sensor sensitivity of the weight sensor beyond athreshold range such that an inaccurate item quantity is determined. Thecurrent item counting technologies are not configured to providetechnical solutions to the above-mentioned technical problems. Thesystem disclosed in the present application provides technical solutionsto the technical problems discussed above by implementing an adaptiveitem counting algorithm that is configured to dynamically adapt the itemcounting process based on weight sensor sensitivity changes and learnfrom historical data records, i.e., items quantities detected indifferent temperatures. In other words, the disclosed system captures ordetects weight sensor drifts (where the sensitivity of the weight sensordrifts beyond a threshold range from the default sensitivity) anddynamically corrects or adjusts the item counting process. With thismethod, more accurate results from the item counting process can beachieved even when the weight sensor sensitivity fluctuates, forexample, due to environmental changes, e.g., temperature changes.

One embodiment of the disclosed system provides several practicalapplications and technical advantages which include: 1) a process fordynamically adapting the item counting process based on weight sensorsensitivity analysis; 2) a process for implementing a patternrecognition technique for cases where a non-integer amount of an item isdetected, for example, such as an item count is calculated to be 2.5;and 3) validating (or updating) the result from the item countingprocess by analyzing an instantaneous (or current) weight sensorsensitivity and a default sensor sensitivity. As such, the disclosedsystem may improve the item counting process by implementing a patternrecognition technique and a validation step based on analyzing theinstantaneous (or current) weight sensor sensitivity.

This disclosure contemplates an unconventional, distributed trackingsystem that can scale to handle larger spaces. The system uses an arrayof cameras, multiple camera clients, a camera server, weight sensors, aweight server, and a central server to determine which person in thespace took an item and should be charged for the item. The cameraclients each process frames of videos from a different subset of camerasof the array of cameras. Each camera client determines coordinates forpeople detected in the frames and then timestamps these coordinatesbased on when the frames were received by the camera client. The cameraclients then communicate the coordinates and timestamps to a cameraserver that is responsible for coordinating the information from thecamera clients. The camera server determines, based on the coordinatesand timestamps from the camera clients, the positions of people in thespace. The weight server processes signals from the weight sensors todetermine when items were removed from shelves in the space. The centralserver uses the positions of people in the space from the camera serverand the determinations from the weight server of when items were removedfrom shelves to determine which people in the space took which items andshould therefore be charged.

Generally, the camera server protects against desynchronization byassigning the coordinates from the multiple camera clients to windows oftime based on the timestamps. The camera server then processes thecoordinates assigned to a particular time window to determine overallcoordinates for people in the space during that time window. Theduration of the time window can be set to be larger than thedesynchronization that is expected to occur to mitigate the effects ofdesynchronization. For example, if the cameras and camera clients areexpected to desynchronize by a few milliseconds, then the time windowcan be set to last 100 milliseconds to counteract the desynchronization.In this manner, the number of cameras and camera clients can beincreased to scale the system to handle any suitable space.

This disclosure also contemplates an unconventional way of wiringcameras in the array of cameras to the camera clients. The cameras arearranged as a rectangular grid above the space. Each camera in the gridis wired to a particular camera client according to certain rules. Forexample, no two cameras that are directly adjacent to one another in thesame row or column of the grid are wired to the same camera client. Asanother example, cameras that are arranged along a diagonal in the gridare wired to the same camera client. In this manner, a small area of thegrid should include cameras that are wired to each and every cameraclient in the system. As a result, even if one camera client were to gooffline (e.g., maintenance, error, or crash), there would still beenough coverage from the remaining camera clients to track the positionsof people in the small area. Thus, this arrangement of the camerasimproves the resiliency of the system.

This disclosure further contemplates an unconventional rack and shelfdesign that integrates weight sensors for tracking when items have beenremoved from the racks and shelves. Generally, the rack includes a base,a vertical panel, and a shelf. The base forms an enclosed space in whicha printed circuit board is positioned, and the base includes a drawerthat opens to provide access to the enclosed space and the circuitboard. The vertical panel is attached to the base, and the shelf isattached to the vertical panel. Weight sensors are positioned within theshelf. The base, panel, and shelf each define a cavity. The cavity inthe shelf and the cavity in the panel are at least partially aligned.Each weight sensor communicates signals to the printed circuit boardthrough a wire that runs from that weight sensor, through the cavity ofthe shelf, the cavity of the panel, and the cavity of the base, to thecircuit board.

Certain embodiments include an unconventional tracking system thatincludes separate components (e.g., camera clients, camera servers,weight servers, and a central server) that perform different functionsto track the positions of people and/or objects in a space. By spreadingthe functionality of the system amongst these various components, thesystem is capable of processing signals from more sensors (e.g., camerasand weight sensors). Due to the increase in the number of sensors, thesystem can track people and/or objects in a larger space. As a result,the system can be scaled to handle larger spaces (e.g., by addingadditional camera clients). Certain embodiments of the tracking systemare described below.

With respect to FIGS. 3A-3T, according to an embodiment, a systemincludes an array of cameras, a first camera client, a second cameraclient, a camera server, a plurality of weight sensors, a weight server,and a central server. The array of cameras is positioned above a space.Each camera of the array of cameras captures a video of a portion of thespace. The space contains a person. The first camera client receives afirst plurality of frames of a first video from a first camera of thearray of cameras. Each frame of the first plurality of frames shows theperson within the space. For a first frame of the first plurality offrames, the first camera client determines a first bounding area aroundthe person shown in the first frame and generates a first timestamp ofwhen the first frame was received by the first camera client. For asecond frame of the first plurality of frames, the first camera clientdetermines a second bounding area around the person shown in the secondframe and generates a second timestamp of when the second frame wasreceived by the first camera client. The second camera client isseparate from the first camera client. The second camera client receivesa second plurality of frames of a second video from a second camera ofthe array of cameras. Each frame of the second plurality of frames showsthe person within the space. For a third frame of the second pluralityof frames, the second camera client determines a third bounding areaaround the person shown in the third frame and generates a thirdtimestamp of when the third frame was received by the second cameraclient. For a fourth frame of the second plurality of frames, the secondcamera client determines a fourth bounding area around the person shownin the fourth frame and generates a fourth timestamp of when the fourthframe was received by the second camera client.

The camera server is separate from the first and second camera clients.The camera server determines that the first timestamp falls within afirst time window and in response to determining that the firsttimestamp falls within the first time window, assigns coordinatesdefining the first bounding area to the first time window. The cameraserver also determines that the second timestamp falls within the firsttime window and in response to determining that the second timestampfalls within the first time window, assigns coordinates defining thesecond bounding area to the first time window. The camera server furtherdetermines that the third timestamp falls within the first time windowand in response to determining that the third timestamp falls within thefirst time window, assigns coordinates defining the third bounding areato the first time window. The camera server determines that the fourthtimestamp falls within a second time window that follows the first timewindow and in response to determining that the fourth timestamp fallswithin the second time window, assigns coordinates defining the fourthbounding area to the second time window.

The camera server also determines that coordinates assigned to the firsttime window should be processed and in response to determining thatcoordinates assigned to the first time window should be processed, thecamera server calculates, based at least on the coordinates defining thefirst bounding area and the coordinates defining the second boundingarea, a combined coordinate for the person during the first time windowfor the first video from the first camera and calculates, based at leaston the coordinates defining the third bounding area, a combinedcoordinate for the person during the first time window for the secondvideo from the second camera. The camera server also determines, basedat least on the combined coordinate for the person during the first timewindow for the first video from the first camera and the combinedcoordinate for the person during the first time window for the secondvideo from the second camera, a position of the person within the spaceduring the first time window.

The plurality of weight sensors are positioned within the space. Eachweight sensor of the plurality of weight sensors produces a signalindicative of a weight experienced by that weight sensor. The weightserver is separate from the first and second camera clients and thecamera server. The weight server determines, based at least on a signalproduced by a first weight sensor of the plurality of weight sensors,that an item positioned above the first weight sensor was removed. Thecentral server is separate from the first and second camera clients, thecamera server, and the weight server. The central server determines,based at least on the position of the person within the space during thefirst time window, that the person removed the item. Based at least onthe determination that the first person removed the item, the person ischarged for the item when the person exits the space.

According to another embodiment, a system includes an array of cameras,a first camera client, a second camera client, a camera server, aplurality of weight sensors, a weight server, and a central server. Thearray of cameras is positioned above a space. Each camera of the arrayof cameras captures a video of a portion of the space. The spacecontains a person. The first camera client, for each frame of a firstvideo received from a first camera of the array of cameras, determines abounding area around the person shown in that frame of the first videoand generates a timestamp of when that frame of the first video wasreceived by the first camera client. The second camera client, for eachframe of a second video received from a second camera of the array ofcameras, determines a bounding area around the person shown in thatframe of the second video and generates a timestamp of when that frameof the second video was received by the second camera client.

The camera server is separate from the first and second camera clients.The camera server, for each frame of the first video, assigns, based atleast on the timestamp of when that frame was received by the firstcamera client, coordinates defining the bounding area around the personshown in that frame to one of a plurality of time windows. For eachframe of the second plurality of frames, the camera server assigns,based at least on the timestamp of when that frame was received by thesecond camera client, coordinates defining the bounding area around theperson shown in that frame to one of the plurality of time windows. Fora first time window of the plurality of time windows, the camera servercalculates, based at least on the coordinates that (1) define boundingareas around the person shown in the first plurality of frames and (2)are assigned to the first time window, a combined coordinate for theperson during the first time window for the first video from the firstcamera and calculates, based at least on the coordinates that (1) definebounding areas around the person shown in the second plurality of framesand (2) are assigned to the first time window, a combined coordinate forthe person during the first time window for the second video from thesecond camera. The camera server determines, based at least on thecombined coordinate for the person during the first time window for thefirst video from the first camera and the combined coordinate for theperson during the first time window for the second video from the secondcamera, a position of the person within the space during the first timewindow.

The plurality of weight sensors are positioned within the space. Theweight server is separate from the first and second camera clients andthe camera server. The weight server determines, based at least on asignal produced by a first weight sensor of the plurality of weightsensors, that an item positioned above the first weight sensor wasremoved. The central server is separate from the first and second cameraclients, the camera server, and the weight server. The central serverdetermines, based at least on the position of the person within thespace during the first time window, that the person removed the item.

Certain embodiments of the tracking system perform an unconventionaltracking process that allows for some desynchronization amongst thecomponents of the system (e.g., camera clients and camera server).Generally, the system processes information according to time windows.These time windows may be set to be larger than the desynchronizationthat is expected to exist in the system. Information that is assigned toa time window is processed together. Thus, even if somedesynchronization exists amongst that information, it is neverthelessprocessed together within the same time window. In this manner, thetracking system can handle an increased amount of desynchronization,especially desynchronization that occurs as a result of the system beingscaled to include more components so that the system can handle a largerspace. As a result, the system can scale to handle larger spaces whilemaintaining reliability and accuracy. Certain embodiments of thetracking process are described below.

According to an embodiment, a system includes an array of cameras, afirst camera client, a second camera client, and a camera server. Thearray of cameras is positioned above a space. Each camera of the arrayof cameras captures a video of a portion of the space. The spacecontains a person. The first camera client receives a first plurality offrames of a first video from a first camera of the array of cameras.Each frame of the first plurality of frames shows the person within thespace. For a first frame of the first plurality of frames, the firstcamera client determines a first bounding area around the person shownin the first frame and generates a first timestamp of when the firstframe was received by the first camera client. For a second frame of thefirst plurality of frames, the first camera client determines a secondbounding area around the person shown in the second frame and generatesa second timestamp of when the second frame was received by the firstcamera client. For a third frame of the first plurality of frames, thefirst camera client determines a third bounding area around the personshown in the third frame and generates a third timestamp of when thethird frame was received by the first camera client.

The second camera client receives a second plurality of frames of asecond video from a second camera of the array of cameras. Each frame ofthe second plurality of frames shows the person within the space. For afourth frame of the second plurality of frames, the second camera clientdetermines a fourth bounding area around the person shown in the fourthframe and generates a fourth timestamp of when the fourth frame wasreceived by the second camera client. For a fifth frame of the secondplurality of frames, the second camera client determines a fifthbounding area around the person shown in the fifth frame and generates afifth timestamp of when the fifth frame was received by the secondcamera client.

The camera server is separate from the first and second camera clients.The camera server determines that the first timestamp falls within afirst time window and in response to determining that the firsttimestamp falls within the first time window, assigns coordinatesdefining the first bounding area to the first time window. The cameraserver also determines that the second timestamp falls within the firsttime window and in response to determining that the second timestampfalls within the first time window, assigns coordinates defining thesecond bounding area to the first time window. The camera server furtherdetermines that the third timestamp falls within a second time windowthat follows the first time window and in response to determining thatthe third timestamp falls within the second time window, assignscoordinates defining the third bounding area to the second time window.The camera server also determines that the fourth timestamp falls withinthe first time window and in response to determining that the fourthtimestamp falls within the first time window, assigns coordinatesdefining the fourth bounding area to the first time window. The cameraserver further determines that the fifth timestamp falls within thesecond time window and in response to determining that the fifthtimestamp falls within the second time window, assigns coordinatesdefining the fifth bounding area to the second time window.

The camera server also determines that coordinates assigned to the firsttime window should be processed and in response to determining thatcoordinates assigned to the first time window should be processed, thecamera server calculates, based at least on the coordinates defining thefirst bounding area and the coordinates defining the second boundingarea, a combined coordinate for the person during the first time windowfor the first video from the first camera and calculates, based at leaston the coordinates defining the fourth bounding area, a combinedcoordinate for the person during the first time window for the secondvideo from the second camera. After determining that coordinatesassigned to the first time window should be processed, the camera serverdetermines that coordinates assigned to the second time window should beprocessed and in response to determining that coordinates assigned tothe second time window should be processed, the camera serverscalculates, based at least on the coordinates defining the thirdbounding area, a combined coordinate for the person during the secondtime window for the first video from the first camera and calculates,based at least on the coordinates defining the fifth bounding area, acombined coordinate for the person during the second time window for thesecond video from the second camera.

According to another embodiment, a system includes an array of cameras,a first camera client, a second camera client, and a camera server. Thearray of cameras is positioned above a space. Each camera of the arrayof cameras captures a video of a portion of the space. The spacecontains a person. The first camera client receives a first plurality offrames of a first video from a first camera of the array of cameras.Each frame of the first plurality of frames shows the person within thespace. For each frame of the first plurality of frames, the first cameraclient determines a bounding area around the person shown in that frameand generates a timestamp of when that frame was received by the firstcamera client. The second camera client receives a second plurality offrames of a second video from a second camera of the array of cameras.Each frame of the second plurality of frames shows the person within thespace. For each frame of the second plurality of frames, the secondcamera client determines a bounding area around the person shown in thatframe and generates a timestamp of when that frame was received by thesecond camera client.

The camera server is separate from the first and second camera clients.The camera server, for each frame of the first plurality of frames,assigns, based at least on the timestamp of when that frame was receivedby the first camera client, coordinates defining the bounding areaaround the person shown in that frame to one of a plurality of timewindows and for each frame of the second plurality of frames, assigns,based at least on the timestamp of when that frame was received by thesecond camera client, coordinates defining the bounding area around theperson shown in that frame to one of the plurality of time windows.

The camera server also determines that coordinates assigned to a firsttime window of the plurality of time windows should be processed and inresponse to determining that coordinates assigned to the first timewindow should be processed, calculates, based at least on thecoordinates that (1) define bounding areas around the person shown inthe first plurality of frames and (2) are assigned to the first timewindow, a combined coordinate for the person during the first timewindow for the first video from the first camera and calculates, basedat least on the coordinates that (1) define bounding areas around theperson shown in the second plurality of frames and (2) are assigned tothe first time window, a combined coordinate for the person during thefirst time window for the second video from the second camera.

Certain embodiments include an unconventional arrangement of cameras andcamera clients that improve the resiliency of the camera system.Generally, the cameras are arranged in a rectangular grid that providescoverage for a physical space, and each camera is communicativelycoupled to one camera client. No camera is directly adjacent in the samerow or column of the grid to another camera that is communicativelycoupled to the same camera client. Cameras arranged along a diagonal ofthe grid are communicatively coupled to the same camera client. In thismanner, even if one camera client in the system were to go offline, thegrid still provides sufficient coverage for the physical space. As aresult, the arrangement of the cameras improves the resiliency of thesystem. Certain embodiments of the camera arrangement are describedbelow.

According to an embodiment, a system includes a first camera client, asecond camera client, a third camera client, and an array of cameras.The second camera client is separate from the first camera client. Thethird camera client is separate from the first and second cameraclients. The array of cameras is positioned above a space. The camerasin the array of cameras are arranged as a rectangular grid comprising afirst row, a second row, a third row, a first column, a second column,and a third column. The array includes first, second, third, fourth,fifth, and sixth cameras.

The first camera is positioned in the first row and the first column ofthe grid. The first camera is communicatively coupled to the firstcamera client. The first camera communicates a video of a first portionof the space to the first camera client. The second camera is positionedin the first row and the second column of the grid such that the secondcamera is directly adjacent to the first camera in the grid. The secondcamera is communicatively coupled to the second camera client. Thesecond camera communicates a video of a second portion of the space tothe second camera client. The third camera is positioned in the firstrow and the third column of the grid such that the third camera isdirectly adjacent to the second camera in the grid. The third camera iscommunicatively coupled to the third camera client. The third cameracommunicates a video of a third portion of the space to the third cameraclient. The fourth camera is positioned in the second row and the firstcolumn of the grid such that the fourth camera is directly adjacent tothe first camera in the grid. The fourth camera is communicativelycoupled to the second camera client. The fourth camera communicates avideo of a fourth portion of the space to the second camera client. Thefifth camera is positioned in the second row and the second column ofthe grid such that the fifth camera is directly adjacent to the fourthcamera and the second camera in the grid. The fifth camera iscommunicatively coupled to the third camera client. The fifth cameracommunicates a video of a fifth portion of the space to the third cameraclient. The sixth camera is positioned in the third row and the firstcolumn of the grid such that the sixth camera is directly adjacent tothe fourth camera in the grid. The sixth camera is communicativelycoupled to the third camera client. The sixth camera communicates avideo of a sixth portion of the space to the third camera client.

According to another embodiment, a system includes a plurality of cameraclients and an array of cameras. The plurality of camera clientsincludes a number of camera clients. The array of cameras is positionedabove a space. Each camera in the array of cameras communicates a videoof a portion of the space to only one camera client of the plurality ofcamera clients. The cameras in the array of cameras are arranged suchthat each camera client of the plurality of camera clients iscommunicatively coupled to at least one camera in an N×N portion of thearray. N is the number of camera clients in the plurality of cameraclients minus one.

With respect to FIGS. 3U-3Y, according to an embodiment, a systemincludes a first camera, a first camera client, and cluster servers. Thefirst camera is configured to generate a first plurality of framesshowing a portion of a space, where the space contains a person. Thefirst camera client is operably coupled with the first camera and isconfigured to receive the first plurality of frames from the firstcamera. The first plurality of frames comprises a first plurality ofcolor frames and a first plurality of depth frames. The first pluralityof color frames corresponds to the visual colors of objects in thespace. The first plurality of depth frames corresponds to distances ofobjects in the space from the first camera. The first camera clientgenerates a timestamp when each corresponding color and depth frame isreceived by the first camera client. The first camera client labels thefirst plurality of color frames and the first plurality of depth frameswith their corresponding timestamps and an identifier number of thefirst camera. The first camera client sends a first set of framescomprising the first plurality of color frames and the first pluralityof depth frames labeled with one or more corresponding timestamps andthe identifier number of the first camera to a first server from among aplurality of cluster servers.

For each depth frame from the first plurality of depth frames, thecamera client detects a contour associated with the person in the depthframe. The camera client detects, based at least in part upon pixelcoordinates of the contour, a bounding area around the person in thedepth frame. The camera client determines, based at least in part uponthe bounding area, coordinates of the person in the depth frame. Thecamera client associates, based at least in part upon the bounding area,a first tracking identification to the person, where the first trackingidentification is linked to historical detections associated with theperson. The historical detections associated with the person comprise atleast one of a contour, a bounding area, a segmentation mask associatedwith the person.

The camera client generates a track comprising the first trackingidentification linked to the historical detections associated with theperson. The track is labeled with one or more corresponding timestamps,the identifier number of the first camera, the first trackingidentification, and the historical detections associated with theperson. The first camera client sends a first set of tracks comprisingtracks generated from the first plurality of depth frames to a secondserver from among the plurality of cluster servers.

Each server from among the plurality of cluster servers is operablycoupled to the first camera and the first camera client, and configuredto store the first set of frames using one or more corresponding labelsof the first set of frames comprising an identifier number of the firstcamera and a timestamp range. Each server from among the plurality ofcluster servers is further configured to store the first set of tracksusing one or more corresponding labels of the first set of trackscomprising the identifier number of the first camera, a timestamp range,the historical detections, and the first tracking identification.

Certain embodiments include an unconventional rack for holding items.The rack includes a base and panels for holding shelves and weightsensors. The weight sensors are wired to a circuit board located in adrawer in the base. The wires run from the weight sensors throughcavities and spaces defined by the shelves, panels, and base. Certainembodiments of the rack are described below.

With respect to FIGS. 5A-5G, according to an embodiment, a systemincludes a circuit board and a rack. The rack includes a base, a panel,a shelf, a first weight sensor, a second weight sensor, a first wire,and a second wire. The base includes a bottom surface, a first sidesurface, a second side surface, a third side surface, a top surface, anda drawer. The first side surface is coupled to the bottom surface of thebase. The first side surface of the base extends upwards from the bottomsurface of the base. The second side surface is coupled to the bottomand first side surfaces of the base. The second side surface of the baseextends upwards from the bottom surface of the base. The third sidesurface is coupled to the bottom and second side surfaces of the base.The third side surface of the base extends upwards from the bottomsurface of the base. The top surface is coupled to the first, second,and third side surfaces of the base such that the bottom and topsurfaces of the base and the first, second, and third side surfaces ofthe base define a space. The top surface of the base defines a firstopening into the space. The drawer is positioned within the space. Thecircuit board is positioned within the drawer.

The panel is coupled to the base and extends upwards from the base. Thepanel defines a second opening that extends along a width of the panel.The shelf is coupled to the panel such that the shelf is positionedvertically higher than the base and such that the shelf extends awayfrom the panel. The shelf includes a bottom surface, a front surfacethat extends upwards from the bottom surface of the shelf, and a backsurface that extends upwards from the bottom surface of the shelf. Theback surface of the shelf is coupled to the panel. The back surface ofthe shelf defines a third opening. A portion of the third opening alignswith a portion of the second opening.

The first weight sensor is coupled to the bottom surface of the shelfand positioned between the front surface of the shelf and the backsurface of the shelf. The second weight sensor is coupled to the bottomsurface of the shelf and positioned between the front surface of theshelf and the back surface of the shelf. The first wire is coupled tothe first weight sensor and the circuit board The first wire extendsfrom the first weight sensor through the second and third openings anddownwards into the space through the first opening. The second wire iscoupled to the second weight sensor and the circuit board. The secondwire extends from the second weight sensor through the second and thirdopenings and downwards into the space through the first opening.

With respect to FIGS. 5K-5M, in one embodiment, a system comprises acircuit board and a rack. The rack includes a base, a panel, a shelf, afirst weight board, a second weight board, a first weight sensor, asecond weight sensor, and a bus wire. The base provides a space toposition the circuit board. The panel is coupled to the base and extendsupwards from the base. The shelf provides a first region to position thefirst weight sensor and a second region to position the second weightsensor.

The first weight board is operably coupled to the first weight sensorand configured to assign a first address number to the first weightsensor corresponding to the first region of the shelf where the firstweight sensor is positioned. The second weight board is operably coupledto the second weight sensor and configured to assign a second addressnumber to the second weight sensor corresponding to the second region ofthe shelf where the second weight sensor is positioned.

The bus wire is used for connecting the first weight board and thesecond weight board to the circuit board. A first wire is connected tothe first weight board and extends from the first weight board outwardsfrom the first region. A second wire is connected to the second weightboard and extends from the second weight board outwards from the secondregion. The first wire and the second wire are aggregated to form thebus wire that connects to a port on the circuit board. The circuit boardis configured to determine from which weight sensor a signal isoriginated based at least in part upon address numbers of the firstweight sensor and the second weight sensor. If the circuit boardreceives a first signal comprising the first address number, the circuitboard determines that the first signal is originated from the firstweight sensor. If the circuit board receives a second signal comprisingthe second address number, the circuit board determines that the secondsignal is originated from the second weight sensor.

With respect to FIGS. 5N and 5O, in one embodiment, a system comprises aweight sensor that is configured to measure a weight of at least one ofan item positioned on the weight sensor, and generate a first signalindicative of the weight of the at least one of the item.

The system also includes a weight server that is operably coupled withthe weight sensor and includes a processor that is configured to detecta first event corresponding to a weight change on the weight sensor whena quantity of the item is removed from the weight sensor. The processordetermines a weight decrease amount corresponding to the weight decreaseon the weight sensor. The processor calculates a result from dividingthe weight decrease amount over a unit weight of the item. The processordetermines an item quantity value by identifying the closest integer tothe result. The processor determines whether the result is within afirst threshold range from the item quantity value. If it is determinedthat the result is within the first threshold range from the itemquantity value, the processor determines that a quantity of the itemwith the amount of the item quantity value is removed from the weightsensor.

If it is determined that the result is not within the first thresholdrange from the item quantity value, the processor uses a patternrecognition technique to determine the item quantity. In this process,the processor compares a weight change pattern of the first signal withweight change patterns from a training dataset comprising a set ofhistorically observed signals, where each signal in the training datasetis associated with a quantity of the item being removed from the weightsensor. The processor finds a particular signal from the trainingdataset whose weight change pattern matches the weight change pattern ofthe first signal. The processor determines that a quantity of the itemassociated with the particular signal is removed from the weight sensor.

Certain embodiments may include none, some, or all of the abovetechnical advantages discussed above. One or more other technicaladvantages may be readily apparent to one skilled in the art from thefigures, descriptions, and claims included herein.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present disclosure, referenceis now made to the following description, taken in conjunction with theaccompanying drawings, in which:

FIG. 1A-1C illustrates an example store that defines a physical space;

FIGS. 2A and 2B illustrate block diagrams of tracking system examplesfor use in the physical store;

FIGS. 3A-3T illustrate an example camera subsystem and its operation inthe tracking system;

FIGS. 3U-3Y illustrate another example camera subsystem and itsoperation in the tracking system;

FIGS. 4A-4D illustrate an example light detection and ranging subsystemand its operation in the tracking system;

FIGS. 5A-5J illustrate an example weight subsystem and its operation inthe tracking system;

FIGS. 5K-5M illustrate another example weight subsystem and itsoperation in the tracking system;

FIG. 5N illustrates an example flow chart of a method for an adaptiveitem counting algorithm based on sensor sensitivity analysis;

FIG. 5O illustrates an example flow chart of a method for validating aresult from an item counting process;

FIGS. 6A-6C illustrate the operation of an example central server foruse in conjunction with the tracking system;

FIG. 7 illustrates an example computer;

FIG. 8 illustrates a diagram of the determination and use ofauto-exclusion zones by the tracking system;

FIG. 9 illustrates an example auto-exclusion zone map generated by thetracking system;

FIG. 10 illustrates a flowchart of an example method of generating andusing auto-exclusion zones for object tracking using the trackingsystem;

FIG. 11 illustrates a diagram of the detection of closely spaced objectsusing the tracking system;

FIG. 12 illustrates a flowchart of an example method of detectingclosely spaced objects using the tracking system;

FIGS. 13A-C illustrate diagrams of the tracking of a person in localimage frames and in the global plane of space 102 using the trackingsystem;

FIGS. 14A-B illustrate the implementation of a particle filter trackerby the tracking system;

FIG. 15 illustrates a flow diagram of an example method of local andglobal object tracking using the tracking system;

FIG. 16 illustrates a diagram of the use of candidate lists for objectidentification during object tracking by the tracking system;

FIG. 17 illustrates a flowchart of an example method of maintainingcandidate lists during object tracking by the tracking system;

FIG. 18 illustrates a diagram of an example tracking subsystem for usein the tracking system;

FIG. 19 illustrates a diagram of the determination of descriptors basedon object features using the tracking system;

FIGS. 20A-C illustrate diagrams of the use of descriptors forre-identification during object tracking by the tracking system; and

FIG. 21 illustrates a flowchart of an example method of objectre-identification during object tracking using the tracking system.

DETAILED DESCRIPTION

Embodiments of the present disclosure and its advantages are bestunderstood by referring to FIGS. 1A through 21 of the drawings, likenumerals being used for like and corresponding parts of the variousdrawings. Additional information is disclosed in U.S. patent applicationSer. No. 16/663,633 entitled “Scalable Position Tracking System ForTracking Position In Large Spaces” (attorney docket no. 090278.0176);U.S. patent application Ser. No. 16/663,415 entitled “Sensor Array ForScalable Position Tracking System” (attorney docket no. 090278.0177;U.S. patent application Ser. No. 17/018,146 entitled “Sensor Array ForScalable Position Tracking System” (attorney docket no. 090278.0250),which is a divisional of U.S. patent application Ser. No. 16/663,415entitled “Sensor Array For Scalable Position Tracking System” (attorneydocket no. 090278.0177); U.S. patent application Ser. No. 16/991,947entitled “Custom Rack For Scalable Position Tracking System” (attorneydocket no. 090278.0247), which is a continuation of U.S. patentapplication Ser. No. 16/663,669 entitled “Custom Rack For ScalablePosition Tracking System” (attorney docket no. 090278.0178), now U.S.Pat. No. 10,783,762; U.S. patent application Ser. No. 16/941,787entitled “Tracking Positions Using a Scalable Position Tracking System”(attorney docket no. 090278.0229), which is a continuation of U.S.patent application Ser. No. 16/663,432 entitled “Tracking PositionsUsing A Scalable Position Tracking System” (attorney docket no.090278.0179), now U.S. Pat. No. 10,769,450; U.S. patent application Ser.No. 16/941,825 entitled “Tracking Positions Using A Scalable PositionTracking System” (attorney docket no. 090278.0230), which is adivisional of U.S. patent application Ser. No. 16/663,432 entitled“Tracking Positions Using A Scalable Position Tracking System” (attorneydocket no. 090278.0179), now U.S. Pat. No. 10,769,450; U.S. patentapplication Ser. No. 16/663,710 entitled “Topview Object Tracking UsingA Sensor Array” (attorney docket no. 090278.0180); U.S. patentapplication Ser. No. 16/663,766 entitled “Detecting Shelf InteractionsUsing A Sensor Array” (attorney docket no. 090278.0181); U.S. patentapplication Ser. No. 16/663,451 entitled “Topview Item Tracking Using ASensor Array” (attorney docket no. 090278.0182); U.S. patent applicationSer. No. 16/663,794 entitled “Detecting And Identifying Misplaced ItemsUsing A Sensor Array” (attorney docket no. 090278.0183); U.S. patentapplication Ser. No. 16/663,822 entitled “Sensor Mapping To A GlobalCoordinate System” (attorney docket no. 090278.0184); U.S. patentapplication Ser. No. 16/941,415 entitled “Sensor Mapping To A GlobalCoordinate System Using A Marker Grid” (attorney docket no.090278.0226), which is a continuation of U.S. patent application Ser.No. 16/794,057 entitled “Sensor Mapping To A Global Coordinate SystemUsing A Marker Grid” (attorney docket no. 090278.0209), now U.S. Pat.No. 10,769,451, which is a continuation of U.S. patent application Ser.No. 16/663,472 entitled “Sensor Mapping To A Global Coordinate SystemUsing A Marker Grid” (attorney docket no. 090278.0185), now U.S. Pat.No. 10,614,318; U.S. patent application Ser. No. 16/663,856 entitled“Shelf Position Calibration In A Global Coordinate System Using A SensorArray” (attorney docket no. 090278.0186); U.S. patent application Ser.No. 16/664,160 entitled “Contour-Based Detection Of Closely SpacedObjects” (attorney docket no. 090278.0189); U.S. patent application Ser.No. 17/071,262 entitled “Action Detection During Image Tracking”(attorney docket no. 090278.0253), which is a continuation of U.S.patent application Ser. No. 16/857,990 entitled “Action Detection DuringImage Tracking” (attorney docket no. 090278.0224), which is acontinuation of U.S. patent application Ser. No. 16/793,998 entitled“Action Detection During Image Tracking” (attorney docket no.090278.0208) now U.S. Pat. No. 10,685,237, which is a continuation ofU.S. patent application Ser. No. 16/663,500 entitled “Action DetectionDuring Image Tracking” (attorney docket no. 090278.0190) now U.S. Pat.No. 10,621,444; U.S. patent application Ser. No. 16/857,990 entitled“Action Detection During Image Tracking” (attorney docket no.090278.0224), which is a continuation of U.S. patent application Ser.No. 16/793,998 entitled “Action Detection During Image Tracking”(attorney docket no. 090278.0208) now U.S. Pat. No. 10,685,237, which isa continuation of U.S. patent application Ser. No. 16/663,500 entitled“Action Detection During Image Tracking” (attorney docket no.090278.0190) now U.S. Pat. No. 10,621,444; U.S. patent application Ser.No. 16/664,219 entitled “Object Re-Identification During Image Tracking”(attorney docket no. 090278.0191); U.S. patent application Ser. No.16/664,269 entitled “Vector-Based Object Re-Identification During ImageTracking” (attorney docket no. 090278.0192); U.S. patent applicationSer. No. 16/664,332 entitled “Image-Based Action Detection Using ContourDilation” (attorney docket no. 090278.0193); U.S. patent applicationSer. No. 16/664,363 entitled “Determining Candidate Object IdentitiesDuring Image Tracking” (attorney docket no. 090278.0194); U.S. patentapplication Ser. No. 16/664,391 entitled “Object Assignment During ImageTracking” (attorney docket no. 090278.0195); U.S. patent applicationSer. No. 16/664,426 entitled “Auto-Exclusion Zone For Contour-BasedObject Detection” (attorney docket no. 090278.0196); U.S. patentapplication Ser. No. 16/884,434 entitled “Multi-Camera Image Tracking OnA Global Plane” (attorney docket no. 090278.0225), which is acontinuation of U.S. patent application Ser. No. 16/663,533 entitled“Multi-Camera Image Tracking On A Global Plane” (attorney docket no.090278.0197) now U.S. Pat. No. 10,789,720; U.S. patent application Ser.No. 16/663,901 entitled “Identifying Non-Uniform Weight Objects Using ASensor Array” (attorney docket no. 090278.0199); U.S. patent applicationSer. No. 16/663,948 entitled “Sensor Mapping To A Global CoordinateSystem Using Homography” (attorney docket no. 090278.0202); U.S. patentapplication Ser. No. 16/664,470 entitled, “Customer-Based Video Feed”(attorney docket no. 090278.0187); and U.S. patent application Ser. No.16/663,710 entitled, “Topview Object Tracking Using a Sensor Array”(attorney docket no. 090278.0180) which are all hereby incorporated byreference herein as if reproduced in their entirety.

Position tracking systems are used to track the physical positions ofpeople and/or objects in a physical space (e.g., a store). These systemstypically use a sensor (e.g., a camera) to detect the presence of aperson and/or object and a computer to determine the physical positionof the person and/or object based on signals from the sensor. In a storesetting, other types of sensors can be installed to track the movementof inventory within the store. For example, weight sensors can beinstalled on racks and shelves to determine when items have been removedfrom those racks and shelves. By tracking both the positions of personsin a store and when items have been removed from shelves, it is possiblefor the computer to determine which user in the store removed the itemand to charge that user for the item without needing to ring up the itemat a register. In other words, the person can walk into the store, takeitems, and leave the store without stopping for the conventionalcheckout process.

For larger physical spaces (e.g., convenience stores and grocerystores), additional sensors can be installed throughout the space totrack the position of people and/or objects as they move about thespace. For example, additional cameras can be added to track positionsin the larger space and additional weight sensors can be added to trackadditional items and shelves. There is a limit, however, to the numberof sensors that can be added before the computing capabilities of thecomputer are reached. As a result, the computing power of the computerlimits the coverage area of the tracking system.

One way to scale these systems to handle larger spaces is to addadditional computers and to divide the sensors amongst these computerssuch that each computer processes signals from a subset of the sensors.However, dividing the sensors amongst multiple computers introducessynchronization issues. For example, the sensors may not communicatesignals to their respective computers at the same time orsimultaneously. As another example, sensors may have different latencieswith their respective computers and thus, it may take more time forsignals from one sensor to reach a computer than signals from anothersensor. As a result, the sensors and computers become desynchronizedwith each other and it becomes more difficult for the computers todetermine, in a cohesive way, the position of persons or objects in thespace and when items were removed.

This disclosure contemplates an unconventional, distributed trackingsystem that can scale to handle larger spaces. The system uses an arrayof cameras, multiple camera clients, a camera server, weight sensors, aweight server, and a central server to determine which person in thespace took an item and should be charged for the item. The cameraclients each process frames of videos from a different subset of camerasof the array of cameras. Each camera client determines coordinates forpeople detected in the frames and then timestamps these coordinatesbased on when the frames were received by the camera client. The cameraclients then communicate the coordinates and timestamps to a cameraserver that is responsible for coordinating the information from thecamera clients. The camera server determines, based on the coordinatesand timestamps from the camera clients, the positions of people in thespace. The weight server processes signals from the weight sensors todetermine when items were removed from shelves in the space. The centralserver uses the positions of people in the space from the camera serverand the determinations from the weight server of when items were removedfrom shelves to determine which people in the space took which items andshould therefore be charged. The system will be described in more detailusing FIGS. 1A-7.

Generally, the camera server protects against desynchronization byassigning the coordinates from the multiple camera clients to windows oftime based on the timestamps. The camera server then processes thecoordinates assigned to a particular time window to determine overallcoordinates for people in the space during that time window. Theduration of the time window can be set to be larger than thedesynchronization that is expected to occur to mitigate the effects ofdesynchronization. For example, if the cameras and camera clients areexpected to desynchronize by a few milliseconds, then the time windowcan be set to last 100 milliseconds to counteract the desynchronization.In this manner, the number of cameras and camera clients can beincreased to scale the system to handle any suitable space. The cameras,camera clients, and camera server will be described in more detail usingFIGS. 1A-3Q.

This disclosure also contemplates an unconventional way of wiringcameras in the array of cameras to the camera clients. The cameras arearranged as a rectangular grid above the space. Each camera in the gridis wired to a particular camera client according to certain rules. Forexample, no two cameras that are directly adjacent to one another in thesame row or column of the grid are wired to the same camera client. Asanother example, cameras that are arranged along a diagonal in the gridare wired to the same camera client. In this manner, a small area of thegrid should include cameras that are wired to each and every cameraclient in the system. As a result, even if one camera client were to gooffline (e.g., maintenance, error, or crash), there would still beenough coverage from the remaining camera clients to track the positionsof people in the area. Thus, this arrangement of the cameras improvesthe resiliency of the system. The camera array will be described in moredetail using FIGS. 3A-3E.

This disclosure also contemplates an unconventional configuration forprocessing frames of videos captured by cameras. Each camera is operablycoupled with a different camera client. Each camera client is migratedto be positioned adjacent to its corresponding camera. Each cameracommunicates frames of videos to its corresponding camera client. Theframes comprise color frames and depth frames. Generally, each cameraclient generates frames and tracks by processing the color frames anddepth frames. The generated frames comprise the color frames and depthframes which are labeled with one or more of an identifier number of thecamera and a timestamp when they arrived at the camera client. Thegenerated tracks comprise metadata that is detected in the depth frames.The metadata comprises one or more of a tracking identification of aperson detected in the depth frames and historical detections associatedwith the person (e.g., a bounding area, a contour, and a segmentationmask). The metadata indicates local detections of the person in thedepth frames. These local detections are later used by the camera serverto determine global detections of the person in the store. The cameraclient labels the tracks with one or more of the trackingidentification, the historical detections, the identifier number of thecamera, and timestamps when they arrived at the camera client. Thecamera client communicates the frames and tracks to cluster servers. Thecluster servers store and maintain the frames and tracks using one ormore of their corresponding labels such that each frame is retrievableusing one or more of its corresponding labels, and each track isretrievable using one or more of its corresponding labels. Thisconfiguration including the cameras, camera clients, and cluster serversis described in more detail using FIGS. 3U-3Y.

This disclosure further contemplates an unconventional rack and shelfdesign that integrates weight sensors for tracking when items have beenremoved from the racks and shelves. Generally, the rack includes a base,a vertical panel, and a shelf. The base forms an enclosed space in whicha printed circuit board is positioned, and the base includes a drawerthat opens to provide access to the enclosed space and the circuitboard. The vertical panel is attached to the base, and the shelf isattached to the vertical panel. Weight sensors are positioned within theshelf. The base, panel, and shelf each define a cavity. The cavity inthe shelf and the cavity in the panel are at least partially aligned.Each weight sensor communicates signals to the printed circuit boardthrough a wire that runs from that weight sensor, through the cavity ofthe shelf, the cavity of the panel, and the cavity of the base, to thecircuit board. The rack and shelf design will be described in moredetail using FIGS. 5A-5K.

This disclosure further contemplates unconventional weight sensors,weight boards, and circuit boards configuration. In this configuration,each weight sensor in a rack is communicatively coupled to a separateweight board that receives signals from its corresponding weight sensorand processes those signals, e.g., digitizes the signals. The weightboards are communicatively coupled with a circuit board that aggregatesthe processed signals. Wires extending from the weight boards areaggregated into a bus wire that connects to a port on the circuit board.Each weight sensor is assigned a particular address number correspondingto its location within the store, such as a shelf number and aregion/zone number. Each weight sensor is configured to send its addressnumber (along with a signal indicative of a weight of an item) to itscorresponding weight board, which is then forwarded to the circuitboard. As such, the location of a weight sensor can be identified fromthe address number of that weight sensor. Therefore, the item positiontracking becomes easier and more accurate when an item is removed fromor put on a weight sensor because when an item is removed from a weightsensor, the location where the item was removed can be determined usingthe address number of the weight sensor. Furthermore, in each rack,signals from a plurality of weight boards are aggregated in a circuitboard. The circuit board communicates the aggregated signals to thecluster servers. In one embodiment, the aggregated signals may beconverted into a transferable format that is compatible with acommunication protocol of the cluster servers. As an example, theaggregated signals may be in a user datagram protocol (UDP) format, andthe cluster servers may accept data in a transmission control protocol(TCP) format. As such, in this example, the circuit board communicatesthe aggregated signals to a UDP/TCP bridge module to convert theUDP-formatted aggregated signals to TCP-format. The UDP/TCP bridgemodule then communicates the TCP-formatted aggregated signals to thecluster servers. The UDP\TCP bridge module may communicate theTCP-formatted aggregated signals to the cluster server, for example, viaa wireless communication, such as WiFi, etc. This configurationincluding weight sensors, weight boards, and circuit boards will bedescribed in more detail using FIGS. 5K-5M.

This disclosure further contemplates unconventional adaptive itemcounting and validation algorithms. The adaptive item counting algorithmis programmed to determine that a quantity of an item is removed from orput on a weight sensor based at least in part upon weight changesdetected by the weight sensor and a pattern recognition technique. Theresult from the adaptive item counting algorithm is validated by thevalidation algorithm that is programmed to validate (or update) theresult based at least in part upon an instantaneous sensor sensitivityand a default sensor sensitivity of the weight sensor. The adaptive itemcounting and validation algorithms are described in FIGS. 5N and 5O.

The system may also include a light detection and ranging (LiDAR)subsystem that will be described in more detail using FIGS. 4A-4D. Thesystem also includes a central server that ties together the camerasubsystem, weight subsystem, and LiDAR subsystem. The central serverwill be described in more detail using FIGS. 6A-6C.

In one embodiment, the tracking system is configured to determine pixelregions from images generated by each sensor which should be excludedduring object tracking. These pixel regions, or “auto-exclusion zones,”may be updated regularly (e.g., during times when there are no peoplemoving through a space). The auto-exclusion zones may be used togenerate a map of the physical portions of the space that are excludedduring tracking. This configuration is described in more detail usingFIGS. 8 through 10.

In one embodiment, the tracking system is configured to distinguishbetween closely spaced people in a space. For instance, when two peopleare standing, or otherwise located, near each other, it may be difficultor impossible for previous systems to distinguish between these people,particularly based on top-view images. In this embodiment, the systemidentifies contours at multiple depths in top-view depth images in orderto individually detect closely spaced objects. This configuration isdescribed in more detail using FIGS. 11 and 12.

In one embodiment, the tracking system is configured to track peopleboth locally (e.g., by tracking pixel positions in images received fromeach sensor) and globally (e.g., by tracking physical positions on aglobal plane corresponding to the physical coordinates in the space).Person tracking may be more reliable when performed both locally andglobally. For example, if a person is “lost” locally (e.g., if a sensorfails to capture a frame and a person is not detected by the sensor),the person may still be tracked globally based on an image from a nearbysensor, an estimated local position of the person determined using alocal tracking algorithm, and/or an estimated global position determinedusing a global tracking algorithm. This configuration is described inmore detail using FIGS. 13A-C through 15.

In one embodiment, the tracking system is configured to maintain arecord, which is referred to in this disclosure as a “candidate list,”of possible person identities, or identifiers (i.e., the usernames,account numbers, etc. of the people being tracked), during tracking. Acandidate list is generated and updated during tracking to establish thepossible identities of each tracked person. Generally, for each possibleidentity or identifier of a tracked person, the candidate list alsoincludes a probability that the identity, or identifier, is believed tobe correct. The candidate list is updated following interactions (e.g.,collisions) between people and in response to other uncertainty events(e.g., a loss of sensor data, imaging errors, intentional trickery,etc.). This configuration is described in more detail using FIGS. 16 and17.

In one embodiment, the tracking system is configured to employ aspecially structured approach for object re-identification when theidentity of a tracked person becomes uncertain or unknown (e.g., basedon the candidate lists described above). For example, rather thanrelying heavily on resource-expensive machine learning-based approachesto re-identify people, “lower-cost” descriptors related to observablecharacteristics (e.g., height, color, width, volume, etc.) of people areused first for person re-identification. “Higher-cost” descriptors(e.g., determined using artificial neural network models) are used whenthe lower-cost descriptors cannot provide reliable results. Forinstance, in some cases, a person may first be re-identified based onhis/her height, hair color, and/or shoe color. However, if thesedescriptors are not sufficient for reliably re-identifying the person(e.g., because other people being tracked have similar characteristics),progressively higher-level approaches may be used (e.g., involvingartificial neural networks that are trained to recognize people) whichmay be more effective at person identification but which generallyinvolve the use of more processing resources. These configurations aredescribed in more detail using FIGS. 18 through 21.

I. System Overview

FIGS. 1A-1D show the tracking system installed in an example storespace. As discussed above, the tracking system may be installed in astore space so that shoppers need not engage in the conventionalcheckout process. Although the example of a store space is used in thisdisclosure, this disclosure contemplates that the tracking system may beinstalled and used in any type of physical space (e.g., a warehouse, astorage center, an amusement park, an airport, an office building,etc.). Generally, the tracking system (or components thereof) is used totrack the positions of people and/or objects within these spaces for anysuitable purpose. For example, at an airport, the tracking system cantrack the positions of travelers and employees for security purposes. Asanother example, at an amusement park, the tracking system can track thepositions of park guests to gauge the popularity of attractions. As yetanother example, at an office building, the tracking system can trackthe positions of employees and staff to monitor their productivitylevels.

FIG. 1A shows an example store 100. Store 100 is a physical space inwhich shoppers can purchase items for sale. As seen in FIG. 1A, store100 is a physical building that includes an entryway through whichshoppers can enter and exit store 100. A tracking system may beinstalled in store 100 so that shoppers need not engage in theconventional checkout process to purchase items from store 100. Thisdisclosure contemplates that store 100 may be any suitable physicalspace. For example, store 100 may be a convenience store or a grocerystore. This disclosure also contemplates that store 100 may not be aphysical building, but a physical space or environment in which shoppersmay shop. For example, store 100 may be a grab and go pantry at anairport, a kiosk in an office building, an outdoor market at a park,etc.

FIG. 1B shows portions of the interior of store 100. As seen in FIG. 1B,store 100 contains shoppers 105, racks 115, and gates 125. Shoppers 105may have entered store 100 through one of gates 125, which allow entryand exit from store 100. Gates 125 prevent shoppers 105 from enteringand/or exiting the store unless gates 125 are opened.

Gates 125 may include scanners 110 and 120. Scanners 110 and 120 mayinclude a QR code scanner, a barcode scanner, or any other suitable typeof scanner that can receive an electronic code embedded withinformation, such as information that uniquely identifies a shopper 105.Shoppers 105 may scan a personal device (e.g., a smartphone) on scanners110 to enter store 100. When a shopper 105 scans a personal device onscanners 110, the personal device may provide scanners 110 an electroniccode that uniquely identifies the shopper 105. When the shopper 105 isidentified and/or authenticated, gate 125 that includes scanner 110opens to allow the shopper 105 into store 100. Each shopper 105 may haveregistered an account with store 100 to receive an identification codefor the personal device.

After entering store 100, shoppers 105 may move around the interior ofstore 100. As the shoppers 105 move throughout the space, shoppers 105may shop for items 130 by removing items 130 from racks 115. As seen inFIG. 1B, store 100 includes racks 115 that hold items 130. When shopper105 wishes to purchase a particular item 130, shopper 105 can removethat item 130 from rack 115. Shoppers 105 can remove multiple items 130from store 100 to purchase those items 130.

When shopper 105 has completed shopping for items 130, shopper 105approaches gates 125. In certain embodiments, gates 125 willautomatically open so that shopper 105 can leave store 100. In otherembodiments, shopper 105 scans a personal device on scanners 120 beforegates 125 will open to allow shopper 105 to exit store 100. When shopper105 scans a personal device on scanner 120, the personal device mayprovide an electronic code that uniquely identifies shopper 105 toindicate that shopper 105 is leaving store 100. When shopper 105 leavesstore 100, an account of shopper 105 is charged for the items 130 thatshopper 105 removed from store 100.

FIG. 1C shows the interior of store 100 along with a tracking system 132that allows shopper 105 to simply leave store 100 with items 130 withoutengaging in a conventional checkout process. As seen in FIG. 1C, thetracking system 132 includes an array of sensors 135 positioned on theceiling of store 100. The array of sensors 135 may provide coverage forthe interior space of store 100. Sensors 135 are arranged in a gridpattern across the ceiling of store 100, as explained in further detailwith respect to FIGS. 3A-3E. Sensors 135 may be used to track thepositions of shoppers 105 within the space of store 100. This disclosurecontemplates sensors 135 being any suitable sensors (e.g., cameras,light detection and range sensors, millimeter wave sensors, etc.).

The tracking system 132 also includes weight sensors 140 positioned onthe racks 115. Weight sensors 140 can detect the weight of items 130positioned on racks 115. When an item 130 is removed from the rack 115,the weight sensor 140 may detect a decrease in weight. The trackingsystem 132 may use that information to determine that a particular item130 was removed from the rack 115.

The tracking system 132 includes a computer system 145. Computer system145 may include multiple computers that operate together to determinewhich shopper 105 took which items 130 from racks 115. The components ofcomputer system 145 and their operation will be described in more detailusing FIGS. 2 through 7. Generally, computer system 145 uses informationfrom sensors 135 and weight sensors 140 to determine which shopper 105removed which items 130 from store 100. In this manner, the shopper 105may be automatically charged for items 130 when the shopper 105 leavesstore 100 through gates 125.

FIG. 2A illustrates a block diagram of an example tracking system 132 afor use in store 100. The example tracking system 132 may be an instanceof tracking system 132 shown in FIG. 1C. As seen in FIG. 2A, thetracking system 132 a includes a camera subsystem 202, a light detectionand range (LiDAR) subsystem 204, and a weight subsystem 206. Thetracking system 132 a includes various sensors 135, such as cameras 205,light detection and range (LiDAR) sensors 210, and weight sensors 215.These sensors 135 are communicatively coupled to various computers of acomputer system 145. For example, the camera subsystem 202 includescameras 205 that are communicatively coupled to one or more cameraclients 220. These camera clients 220 are communicatively coupled to acamera server 225. The LiDAR subsystem 204 includes LiDAR sensors 210that are communicatively coupled to a LiDAR server 230. The weightsubsystem 206 includes weight sensors 215 that are communicativelycoupled to a weight server 235. The camera server 225, LiDAR server 230,and weight server 235 are communicatively coupled to a central server240.

Generally, cameras 205 produce videos of portions of the interior of aspace. These videos may include frames or images of shoppers 105 withinthe space. The camera clients 220 process the frames from the cameras205 to detect shoppers 105 within the frames and to assign framecoordinates to those shoppers 105. The camera server 225 generallyprocesses frame data from the camera clients 220 to determine thephysical position of shoppers 105 within the space. LiDAR sensors 210generally produce coordinates of shoppers 105 within a space. LiDARserver 230 processes these coordinates to determine the position ofshoppers 105 within a space. Weight sensors 215 detect the weight ofitems 130 on racks 115 within the space. Weight server 235 processesthese weights to determine when certain items 130 have been removed fromthe racks 115.

Central server 240 processes position information for shoppers 105 fromcamera server 225 and LiDAR server 230 and weight information fromweight server 235 to determine which shopper 105 removed which items 130from the racks 115. These shoppers 105 may then be charged for thoseitems 130 when the shoppers 105 leave the space. The operation of thesecomponents will be described in more detail using FIGS. 3A through 6C.

In one embodiment, each of the components of tracking system 132 a (e.g.camera clients 220, camera server 225, LiDAR server 230, weight server235, and central server 240) is a distinct computing device separatefrom the other components of tracking system 132 a. For example, each ofthese components may include its own processor, memory, and physicalhousing. In this manner, the components of tracking system 132 a aredistributed to provide additional computing power relative to a trackingsystem that includes only one computer.

FIG. 2B illustrates a block diagram of an example tracking system 132 bfor use in the store 100. The example tracking system 132 b may be aninstance of the tracking system 132 shown in FIG. 1C. The exampletracking system 132 b may share one or more components with the trackingsystem 132 a of FIG. 2A. As seen in FIG. 2B, the tracking system 132 bincludes a camera subsystem 242, LiDAR subsystem 204, weight subsystem250, and cluster servers 248.

The camera subsystem 242 may include cameras 205 and camera clients 220,where each camera 205 is communicatively coupled with a separate cameraclient 220. The camera clients 220 are communicatively coupled with thecluster servers 248. The camera subsystem 242 is described in detail inFIGS. 3U-3Y.

The LiDAR subsystem 204 includes LiDAR sensors 210 that arecommunicatively coupled to the LiDAR server 230. The LiDAR server 230 iscommunicatively coupled with the cluster servers 248. The LiDARsubsystem 204 is described in detail in FIGS. 4A-4D.

The weight subsystem 250 may include weight sensors 215 and weightboards 252, where each weight sensor 215 is communicatively coupled witha separate weight board 252. The weight boards 252 are communicativelycoupled with a circuit board 254 that is communicatively coupled withthe cluster servers 248. The weight subsystem 250 is described in detailin FIGS. 5K-5M.

Details of the operations of the cluster servers 248 are described incorresponding descriptions of FIGS. 3U-3W. In brief, the cluster servers248 may be implemented by a plurality of servers or computing devices,where each server is configured to store and maintain data based on oneor more corresponding labels or indexes associated with that data. Assuch, each server from the cluster servers 248 is able to return dataupon receiving a request to retrieve that data using one or more of itscorresponding labels. In other words, data stored in the cluster servers248 is retrievable or query-searchable using one or more of itscorresponding labels. Each server from the cluster servers 248 is alsoconfigured to provide streaming a particular set of data (e.g., frames)upon receiving a request to stream the particular set of data (e.g.,frames).

In the illustrated tracking system 132 b, the cluster servers 248 areconfigured to receive data from the camera clients 220, LiDAR server230, and circuit board 254; and publish or transmit data to a computingdevice that is subscribed to the cluster servers 248, such as the cameraserver 225, weight server 235, and central server 240. For example, thecamera clients 220 may send a first set of data (e.g., frames 314) to afirst server from the cluster servers 248; and a second set of data(e.g., tracks 316) to a second server from the cluster servers 248. Seecorresponding descriptions of FIGS. 3V and 3. W for explanations offrames 314 and tracks 316. In brief, frames 314 include color frames 364and depth frames 366 which are labeled with their correspondingtimestamps and an identifier number of a camera 305 they were capturedfrom. Tracks 316 include metadata associated with the depth frames 366.The tracks 316 include historical detection of people detected in thedepth frames 366 (e.g., bounding areas, contours, segmentation masks,etc.), tracking identifications 386 of people detected in the depthframes 366, timestamps of depth frames 366, among others.

For example, the camera server 225 may subscribe to the cluster servers248 and send a request to the first server from the cluster servers 248to transmit the first set of data (e.g., frames 314) for processing.Similarly, the camera server 225 may send a request to the second serverfrom the cluster servers 248 to transmit the second set of data (e.g.,tracks 316) for processing. In another example, the circuit board 254may send a third set of data (e.g., weight information of items) to athird server from the cluster servers 248. For example, the weightserver 235 may subscribe to the cluster servers 248 and send a requestto the third server from the cluster servers 248 to transmit the thirdset of data for processing.

In one embodiment, data coming from the circuit board 254 may beconverted into a transferrable format that is compatible with acommunication protocol of the cluster servers 248. For example, theincoming data from the circuit board 254 may be in a user datagramprotocol (UDP) format. In order to forward data coming from the circuitboard 254 to the cluster servers 248, it may be converted into atransferrable format that is compatible with the cluster servers 248,such as a transmission control protocol (TCP) format. As such, a UDP/TCPbridge module (not shown) that bridges the UDP communication protocoland the TCP communication protocol may be used to convert incomingUDP-formatted data into TCP-formatted data. In one embodiment, thecircuit board 565 may transmit data to the UDP/TCP bridge module througha wireless communication, such as WiFi or any other method of wirelesscommunication. The UDP/TCP bridge module forwards the data coming fromthe circuit board 254 in the TCP format to the cluster servers 248. In aparticular example, cluster servers 248 may be implemented in a Kafkacluster framework where one or servers in the Kafka cluster frameworkare configured to store, maintain, and stream data.

II. Camera Subsystem

FIGS. 3A-3R show an example camera subsystem 202 and its operation inthe tracking system 132. As discussed above, the camera subsystem 202includes cameras 205, camera clients 220, and a camera server 225.Generally, the cameras 205 capture video of a space and send the videosto the camera clients 220 for processing. These videos are a sequence offrames or images of the space. The camera clients 220 detect thepresence of people (e.g., shoppers 105) in the frames and determinecoordinates in the frames (may also be referred to as “framecoordinates”) for those people. The camera server 225 analyzes the framecoordinates from each camera client 220 to determine physical positionsof the people in the space.

1. Camera Array

FIG. 3A illustrates an example camera array 300. As shown in FIG. 3A,camera array 300 includes multiple cameras 305. Although this disclosureshows camera array 300 including twelve cameras 305, camera array 300may include any suitable number of cameras 305. Generally, camera array300 is positioned above a space so that cameras 305 can capture overheadvideos of portions of the space. These videos may then be processed byother components of the camera subsystem 202 to determine the physicalposition of people (e.g., shoppers 105) within the space. In the exampleof FIG. 3A, camera array 300 includes cameras 305A, 305B, 305C, 305D,305E, 305F, 305G, 305H, 305I, 305J, 305K, and 305L.

Generally, cameras 305 in camera array 300 are arranged to form arectangular array. In the example of FIG. 3A, camera array 300 is a 3×4array of cameras 305 (e.g., three rows and four columns of cameras 305).Camera array 300 may include any suitable number of cameras 305 arrangedin an array of any suitable dimensions.

Each camera 305 of camera array 300 is communicatively coupled to acamera client 220. In the example of FIG. 3A, each camera 305 of cameraarray 300 is communicatively coupled to one of camera client 1 220A,camera client 2 220B, or camera client 3 220C. Each camera 305communicates captured video to the camera client 220 to which the camera305 is communicatively coupled. The cameras 305 are communicativelycoupled to the camera clients 220 according to particular rules toimprove the resiliency of the tracking system 132. Generally, thecameras 305 are communicatively coupled to the camera clients 220 sothat even if one camera client 220 goes offline, the coverage of aphysical space provided by the cameras 305 communicatively coupled tothe remaining camera clients 220 is sufficient to allow the trackingsystem 132 to continue tracking the position of people within the space.

Cameras 305 are communicatively to camera clients 220 using any suitablemedium. For example, cameras 305 may be hardwired to camera clients 220.As another example, cameras 305 may wirelessly couple to camera clients220 using any suitable wireless protocol (e.g., WiFi). Cameras 305communicate captured videos through the communication medium to thecamera clients 220.

Cameras 305 may be any suitable devices for capturing videos of thespace. For example, cameras 305 may be three-dimensional cameras thatcan capture two-dimensional video of the space (e.g., x-y plane) andalso detect the heights of people and/or objects in the video (e.g., zplane). As another example, cameras 305 may be two-dimensional camerasthat capture two-dimensional videos of the space. Camera array 300 mayinclude a mixture of different types of cameras 305.

FIG. 3B illustrates the coverage provided by cameras 305 of a cameraarray 300. As seen in FIG. 3B, a floor space is covered by differentfields of view 310. Each field of view 310 is provided by a camera 305of camera array 300. For example, field of view 310A is provided bycamera 305A. Field of view 310B is provided by camera 305B. Field ofview 310C is provided by camera 305C, and so forth. Each field of view310 is generally rectangular in shape and covers a portion of the floorspace. Each camera 305 captures video of the portion of the floor spacethat is covered by that camera's 305 field of view 310. For example,camera 305A captures video of the portion of the floor space covered byfield of view 310A. Camera 305B captures video of the portion of thefloor space covered by field of view 310B. Camera 305C captures video ofthe portion of the floor space covered by field of 310C, and so forth.

Each field of view 310 is shaded differently than its neighbors todistinguish the fields of view 310. Fields of view 310A, 310C, 310I, and310K are shaded using lines that slant downwards to the right. Fields ofview 310B, 310D, 310J, and 310L are shaded using lines that slantupwards to the right. Fields of view 310E and 310G are shaded usinghorizontal lines, and fields of view 310F and 310H are shaded usingvertical lines. The shading of each field of view 310 is meant todistinguish that field of view 310 from other, directly adjacent fieldsof view 310. The shading is not meant to indicate a particularcharacteristic of the field of view 310. In other words, even thoughcertain fields of view 310 share the same shading, the similar shadingdoes not indicate that these fields of view 310 share certaincharacteristics (e.g., size, coverage, duration, and/or shape). Fieldsof view 310 may share one or more of these characteristics irrespectiveof their individual shading.

As seen in FIG. 3B, each field of view 310 overlaps with other fields ofview 310. For example, field of view 310A overlaps fields of view 310B,310E, and 310F. As another example, field of view 310F overlaps withfields of view 310A, 310B, 310C, 310E, 310G, 310I, 310J, and 310K. Likefields of view 310A and 310F, other fields of view 310 (e.g., fields ofview 310B, 310C, 310D, 310E, 310G, 310H, 310I, 310J, 310K, and 310L)also overlap neighboring fields of view 310. The shading in theoverlapping regions is a combination of the shadings in the individualfields of view that form the overlapping regions. For example, theoverlapping region formed by fields of view 310A and 310B includesslanted lines running in opposite directions. As another example, theoverlapping region formed by fields of view 310A, 310B, 310E, and 310Fincludes slanted lines running in opposite directions, horizontal lines,and vertical lines.

The overlapping fields of view 310 may be a result of the proximity ofcameras 305 to each other in camera array 300. Generally, by overlappingfields of view 310, certain portions of the floor space can be capturedby multiple cameras 305 of the camera array 300. As a result, even ifcertain cameras 305 go offline, there may still be sufficient coverageprovided by the remaining cameras 305 for the tracking system 132 tooperate. Additionally, the overlapping fields of view 310 may improvetracking the positions of people (e.g., shoppers 105) as they move aboutthe space.

FIG. 3C illustrates an example camera grid 315. As seen in FIG. 3C,camera grid 315 includes a number of rows and a number of columnscorresponding to the number of rows and columns in camera array 300.Each box of camera grid 315 represents a camera 305 of camera array 300.Camera grid 315 shows how the cameras 305 of camera array 300 arecommunicatively coupled to camera clients 220. Using the previousexample of FIG. 3A, camera grid 315 shows that cameras 305A, 305D, 305G,and 305I are communicatively coupled to camera client 1 220A. Cameragrid 315 also shows that cameras 305B, 305E, 305H, and 305K arecommunicatively coupled to camera client 2 220B. Camera grid 315 furthershows that cameras 305C, 305F, 305I, and 305L are communicativelycoupled to camera client 3 220C.

Camera grid 315 shows that cameras 305 are communicatively coupled tocamera clients 220 according to particular rules. For example, a camera305 that is communicatively coupled to a particular camera client 220 isnot directly adjacent in the same row or the same column of camera grid315 to another camera 305 that is communicatively coupled to the samecamera client 220. As seen in FIG. 3C, for example, camera 305A isdirectly adjacent in the same row or the same column of camera grid 315to cameras 305B and 305E. Camera 305A is communicatively coupled tocamera client 1 220A while cameras 305B and 305E are communicativelycoupled to camera client 2 220B. Camera 305F is directly adjacent in thesame row or the same column of camera grid 315 to cameras 305B, 305E,305G, and 305J. Camera 305F is communicatively to camera client 3 220C,while cameras 305B, 305E, 305G, and 305I are communicatively coupled tocamera client 1 220A or camera client 2 220B.

As another example, a camera 305 that is communicatively coupled to aparticular camera client 220 is diagonal in camera grid 315 to anothercamera 305 that is communicatively coupled to the same camera client220. As seen in FIG. 3C, for example, cameras 305D, 305G, and 305J arediagonal to each other and are communicatively coupled to camera client1 220A. Cameras 305C, 305F, and 305I are diagonal to each other and areall communicatively coupled to camera client 3 220C.

A consequence of arranging cameras 305 in this manner is that eachcamera client 220 is communicatively coupled to at least one camera 305in a portion of camera grid 315. As seen in the example of FIG. 3C, eachof camera client 1 220A, camera client 2 220B, and camera client 3 220Cis communicatively coupled to at least one camera in any 2×2 portion ofcamera grid 315. As a result, even if one camera client 220 were to gooffline, the other cameras in the 2×2 portion can still providesufficient coverage of that 2×2 portion to allow the tracking system 132to operate. Thus, the resiliency of the tracking system 132 is improved.

Although the previous example used a certain number of cameras 305 and acertain number of camera clients 220, the tracking system 132 may useany suitable number of cameras 305 and any suitable number of cameraclients 220 to provide a desired level of overlap, scalability, andresiliency. FIG. 3D shows an example camera array 300 that includesadditional cameras 305. The example of FIG. 3D also includes additionalcamera clients 220: camera client 1 220A through camera client N 220D.The cameras 305 in camera array 300 may be communicatively coupled tocamera clients 220 according to the same rules or principles describedin FIGS. 3A through 3C.

FIG. 3E shows how the cameras 305 may communicatively couple to thecamera clients 220. As seen in FIG. 3E, camera grid 315 includes anumber of rows and a number of columns. Across a row, the cameras 305are communicatively coupled to the camera clients 220 in a sequentialfashion. After a camera 305 is communicatively coupled to camera clientN 220 d, the sequence repeats until the end of the row is reached.Similarly, the cameras 305 in a column are sequentially coupled tocamera clients 220. After a camera 305 is communicatively coupled tocamera client N 220 d, the pattern repeats.

As shown in FIGS. 3D and 3E, the tracking system 132 may be scaled toinclude any number of cameras 305 and any number of camera clients 220.Generally, a camera 305 that is communicatively coupled to a particularcamera client 220 is not directly adjacent in the same row or the samecolumn of camera grid 315 to another camera 305 that is communicativelycoupled to the same camera client 220. Additionally, cameras 305 along adiagonal of camera grid 315 are communicatively coupled to the samecamera client 220. Furthermore, each camera client 220 iscommunicatively coupled to at least one camera 305 in a portion ofcamera grid 315. The dimensions of the portion may depend upon thenumber of camera clients 220 in the tracking system 132. Generally, thedimensions of the portion are one less than the number of camera clients220 in the tracking system 132. So, in the examples of FIGS. 3D and 3E,the dimensions of the portion are (N−1)×(N−1).

2. Initialization

FIG. 3F shows the initialization of the camera subsystem 202. As seen inFIG. 3F, the camera subsystem 202 includes a camera array 300, cameraclient 1 220A, camera client 2 220B, camera client 3 220C, and cameraserver 225. Camera subsystem 202, may include any suitable number ofcamera arrays 300, camera clients 220, and camera servers 225.Generally, during initialization, the cameras 305 of the camera array300 start up and begin sending videos 302 to camera clients 220.Additionally, camera clients 220 and camera server 225 synchronizeinternal clocks 304. After the cameras 305 in camera array 300 havestarted up and after the internal clocks 304 are synchronized, cameraclients 220 may begin processing videos 302 and communicatinginformation to camera server 225 to perform the tracking operations ofthe camera subsystem 202.

During initialization, the cameras 305 of camera array 300 may power onand perform a startup sequence. For example, the components of thecameras 305 may boot up and/or warm-up. The cameras 305 may then begincapturing video footage and communicating videos 302 to their respectivecamera clients 220. The cameras 305 of camera array 300 may takedifferent amounts of time to initialize. For example, certain cameras305 may take a shorter or longer amount of time to initialize than othercameras 305 of camera array 300. Because the cameras 305 of camera array300 do not wait for the other cameras 305 of camera array 300 tocomplete initialization before sending videos 302 to camera clients 220,the cameras 305 of camera array 300 may each begin sending videos 302 tocamera clients 220 at different times. As a result, videos 302, and inparticular, the frames of videos 302, may be desynchronized from theframes of other videos 302. In other words, the frames of these videos302 are not being captured and sent by their respective cameras 305simultaneously or at the same time. Consequentially, the frames of thesevideos 302 do not arrive at the camera clients 220 simultaneously or atthe same time.

During initialization, camera clients 220 and camera server 225 power onand/or perform a bootup sequence. After booting up, camera clients 220and camera server 225 synchronize their internal clocks 304. In theexample of FIG. 3F, camera client 1 220A has an internal clock 1 304A.Camera client 2 220B has an internal clock 2 304B. Camera client 3 220Chas an internal clock 3 304C. Camera server 225 has an internal clock 4304D. Camera clients 220 and camera server 225 may synchronize theirinternal clocks 304 in any suitable manner. For example, camera clients220 and camera server 225 may synchronize their internal clocks 304using a synchronization protocol, such as the Network Time Protocol(NTP) or the Precision Time Protocol (PTP). Although a synchronizationprotocol may be used to synchronize the internal clocks 304 of cameraclients 220 and camera server 225, this does not mean that theseinternal clocks 304 show exactly the same time or are perfectlysynchronized with each other. As a result, there may still be a level ofdesynchronization amongst camera clients 220 and camera server 225.

Camera clients 220 may track the cameras 305 of camera array 300 thathave completed initialization by tracking which cameras 305 havecommunicated videos 302 to camera clients 220. When camera clients 220determine that each camera 305 of camera array 300 have begun sendingvideos 302 to camera clients 220, camera clients 220 may determine thatcamera array 300 has finished initialization. In response to thatdetermination, camera clients 220 may begin processing the frames of thevideos 302 and communicating information from those frames to cameraserver 225. Camera server 225 may then analyze the information fromcamera clients 220 to determine the physical position of people and/orobjects within a space.

3. Camera Clients

FIGS. 3G-3I show the operation of camera clients 220 in the camerasubsystem 202. Generally, camera clients 320 process videos 302 fromcameras 305. Camera clients 320 may identify people or objects withinthe frames 320 of these videos 302 and determine coordinates 322 forthese people or objects. Camera clients 320 may also generate timestamps324 (e.g., by using internal clocks 304) that indicate when the cameraclients 320 received particular frames 320. Camera clients 320communicate these timestamps 324 and coordinates 322 to camera server225 for further processing.

FIGS. 3G-3I show the operation of camera clients 210 as an event in astore 100 unfolds. During this event, for example, a first shopper 105(e.g., a man) removes an item 130 from a shelf in the store 100 and asecond shopper 105 (e.g., a woman) moves towards the shelf. Cameraclients 320 analyze frames 320 of videos 302 to determine coordinates322 for the man and the woman in the frames 320.

As seen in FIG. 3G, a man is standing near a shelf and a woman isstanding further away from the shelf. Two cameras 305A and 305B arepositioned above the space and capture video 302 of the man and thewoman and the shelf. These cameras 305A and 305B send their videos 302to two different camera clients 220A and 220B. Camera 305A sends video305 to camera client 220A. Camera 305B sends video 305 to camera client220B.

Camera client 220A receives video 305 from camera 305A, and specificallya frame 320A of that video 305. Camera client 220A processes the frame320A. As seen in frame 320A, the man is standing near the shelf and thewoman is standing further away from the shelf. Camera client 220Aprocesses frame 320A to determine bounding areas 325A and 325B aroundthe man and the woman. In the example of FIG. 3G, bounding areas 325Aand 325B are rectangular areas that surround the man and the woman,respectively. Bounding areas 325A and 325B approximate the positions ofthe man and the woman in the frame. This disclosure contemplates cameraclients 220 determining bounding areas 325 that are of any suitableshape and of any suitable size. For example, bounding areas 325 may becircular or may be irregularly shaped (e.g, so as to follow the contoursof the shopper 105 in the frames 320).

Camera client 220A determines coordinates 322 that define the boundingareas 325A and 325B within frames 320A and 320B (also referred to as“frame coordinates”). In the example of FIG. 3G, camera client 228determines coordinates 322 (x₁, y₁) and (x₂, y₂) for bounding area 325Aand coordinates 322 (x₃, y₃) and (x₄, y₄) for bounding area 325B. Thesecoordinates 322 do not represent absolute coordinates in the physicalspace, but rather coordinates within the frame 320A. Camera clients 220may determine any suitable number of coordinates 322 for bounding areas325.

Camera client 220A then generates frame data 330A that containsinformation about frame 320A. As seen in FIG. 3G, frame data 330Aincludes an identifier for camera 305A (e.g., “camera=1”). Camera client220A may also generate a timestamp 324 (e.g., using internal clock 304)that indicates when frame 320A was received by camera client 220A. Inthe example of FIG. 3G, that timestamp 324 is t₁. Frame data 320A alsoincludes information about the people or objects within frame 320A. Inthe example of FIG. 3G, frame data 330A includes information for anobject 1 and an object 2. Object 1 corresponds to the man and object 2corresponds to the woman. Frame data 330A indicates the coordinates 322for the man (x₁, y₁) and (x₂, y₂) along with a height of the man z₁. Asdiscussed previously, cameras 305 may be three-dimensional cameras thatcan detect the height of objects and/or people. Cameras 305 may haveprovided the heights of the man and the woman to the camera clients 320.In the example of FIG. 3G, camera 305A may have detected the heights ofthe man and the woman to be z₁ and z₂, respectively. Frame data 330Aalso includes information for the woman including the coordinates 322(x₃, y₃) and (x₄, y₄) and the height z₂. Camera client 220A maycommunicate frame data 330A to camera server 225 when frame data 330A isready.

In a corresponding manner, camera client 220B may process video 302 fromcamera 305B. As seen in FIG. 3G, camera client 220B receives a frame320B from camera 305B. Because camera 305B is at a different positionthan camera 305A, frame 320B will show a slightly different perspectiveof the event in the store 100 than frame 320A. Camera client 220Bdetermines bounding areas 325C and 325D around the man and the woman,respectively. Camera client 220B determines frame coordinates 322 (x₁,y₁) and (x₂, y₂) for bounding area 325C, and frame coordinates 322 (x₃,y₃) and (x₄, y₄) for bounding area 325D. Camera client 220B alsodetermines and generates a timestamp 324 t₂ (e.g., using internal clock304) that indicates when camera client 220B received frame 320B. Cameraclient 220B then generates frame data 330B for frame 320B. Frame data330B indicates that frame 320B was generated by camera 305B and wasreceived by camera client 220B at t₂. Frame data 330B also indicatesthat a man and a woman were detected in frame 320B. The man correspondsto coordinates 322 (x₁, y₁) and (x₂, y₂) and has a height z₁. The womancorresponds to coordinates 322 (x₃, y₃) and (x₄, y₄) and has a heightz₂. Camera client 220B communicates frame data 320B to camera server 225when frame data 320B is ready.

The coordinates 322 generated by camera clients 220A and 220B for framedata 330A and 330B may be the coordinates within a particular frame 320and not the coordinates within the physical space. Additionally,although the same subscripts have been used for the coordinates 322 inframe data 330A and 330B, this does not mean that these coordinates 322are the same. Rather, because cameras 305A and 305B are in differentpositions, it is likely that the coordinates 322 in frame 330A aredifferent from the coordinates 322 in frame data 330B. Camera clients220A and 220B are determining the coordinates 322 of the bounding areas325 within the frames 320 and not within the physical space. Cameraclients 220A and 220B determine these local coordinates 322independently of each other. The subscripts indicate a sequence ofcoordinates 322 generated by the individual camera clients 220. Forexample (x₁, y₁) indicates the first coordinate 322 generated by cameraclient 220A and the first coordinate 322 generated by camera client220B, which may be different values.

In FIG. 3H, the event in the store 100 has progressed. The man is stillstanding by the shelf and the woman has moved closer to the shelf.Camera clients 220A and 220B receive additional frames 320C and 320Dfrom cameras 305A and 305B. Camera client 220A again determines boundingareas 325C and 325D for the man and the woman, respectively, andcoordinates 322 for these bounding areas 325. Camera client 220Adetermines coordinates 322 (x₅, y₅) and (x₆, y₆) for bounding area 325Cand coordinates 322 (x₇, y₇) and (x₈, y₈) for bounding area 325D. Cameraclient 220A also generates a timestamp 324 that indicates that frame320C was received at time t₃. Camera client 220A generates frame data330C, indicating that frame 320C was generated by camera 305A andreceived by camera client 220A at t₃. Frame data 330C also indicatesthat the man corresponds to coordinates 322 (x₅, y₅) and (x₆, y₆) andhas a height at z₃ within frame 320C and that the woman corresponds tocoordinates 322 (x₇, y₇) and (x₈, y₈) and has a height at z₄ withinframe 320C.

Similarly, camera client 220B receives frame 320D from camera 305B.Camera client 220B determines bounding areas 325E and 325F for the manand the woman, respectively. Camera client 220B then determinescoordinates 322 (x₅, y₅) and (x₆, y₆) for bounding area 325E andcoordinates 322 (x₇, y₇) and (x₈, y₈) for bounding area 325F. Cameraclient 220B generates a timestamp 324 that indicates that frame 320D wasreceived at time t₄. Camera client 220B generates frame data 330D thatindicates frame 320D was generated by camera 305B and received by cameraclient 220B at t₄. Frame data 330D indicates that the man corresponds tocoordinates 322 (x₅, y₅) and (x₆, y₆) and has a height of z₃ in frame320D. Frame data 330D also indicates that the woman corresponds tocoordinates 322 (x₇, y₇) and (x₈, y₈) and has a height of z₄ withinframe 320D. Camera clients 220A and 220B communicate frame data 330C and330D to camera sever 225 when frame data 330C and 330D are ready.

In FIG. 3I, the event in the store 100 has further progressed and theman has removed an item 130 from the shelf. Camera client 220A receivesa frame 320E from camera 305A. Camera client 220A determines boundingareas 325G and 325H around the man and the woman, respectively. Cameraclient 220A determines coordinates 322 (x₉, y₉) and (x₁₀, y₁₀) forbounding area 325G and coordinates 322 (x₁₁, y₁₁) and (x₁₂, y₁₂) forbounding area 325H. Camera client 220A generates a timestamp 324indicating when frame 320E was received by camera client 220A (e.g., byusing internal clock 304). Camera client 220A generates frame data 330Ethat indicates that frame 320E was produced by camera 305A and receivedby camera client 220A at t₅. Frame data 330E indicates that the mancorresponds to coordinates 322 (x₉, y₉) and (x₁₀, y₁₀) and has a heightat z₅ within frame 320E. Frame data 330E also indicates that the womancorresponds to coordinates 322 (x₁₁, y₁₁) and (x₁₂, y₁₂) and has aheight at z₆ in frame 320E.

Camera client 220B receives frame 320F from camera 305B. Camera client220B determines bounding areas 3251 and 325J around the man and thewoman, respectively. Camera client 220BA determines coordinates 322 (x₉,y₉) and (x₁₀, y₁₀) for bounding area 3251 and coordinates 322 (x₁₁, y₁₁)and (x₁₂, y₁₂) for bounding area 325J. Camera client 220B generates atimestamp 324 indicating when frame 320F was received by camera client220B (e.g., by using internal clock 304). Camera client 220B thengenerates frame data 330F indicating that frame 320F was produced bycamera 305B and received by camera client 220B at t₆. Frame data 330Findicates that the man corresponds to coordinates 322 (x₉, y₉) and (x₁₀,y₁₀) and has a height at z₅ in frame 320F. Frame data 330F alsoindicates that the woman corresponds to coordinates 322 (x₁₁, y₁₁) and(x₁₂, y₁₂) and has a height at z₆ in frame 320F. Camera clients 220A and220B communicate frame data 330E and 330F to camera server 225 whenready.

4. Camera Server

FIGS. 3J-3P show the operation of camera server 225 in the camerasubsystem 202. Generally, camera server 225 receives frame data 330(e.g., 330A-330F) from the camera clients 220 in camera subsystem 202.Camera server 225 synchronizes and/or assigns the frame data 330 toparticular time windows 332 based on timestamps 324 in the frame data330. Camera server 225 then processes the information assigned toparticular time windows to determine the physical positions of peopleand/or objects within the space during those time windows 332.

In FIG. 3J, camera server 225 receives frame data 330 from the cameraclients 220 in camera subsystem 202. Camera server 225 assigns framedata 330 to time windows 332 depending on the timestamp 324 within framedata 330. Using the previous example, camera server 225 may determinethat timestamps 324 t₁, t₂, and t₃ fall within a first time window 322A(e.g., between times T0 and T1) and that timestamps 324 t₄, t₅, and t₆fall within a subsequent time window 332B (e.g., between times T1 andT2). As a result, camera server 225 assigns the frame data 330 forframes 320A, 320B, and 320C to time window 1 332A and the frame data 330for frames 320D, 320E, and 320F to time window 2 332B.

By assigning frame data 330 to time windows 332, camera server 225 mayaccount for desynchronization that occurs amongst the cameras 305,camera clients 220, and the camera server 225 in the camera subsystem202. The duration of the time windows 332 can be set to be larger thanthe desynchronization that is expected to occur to mitigate the effectsof desynchronization. For example, if the cameras 305 and camera clients220 are expected to desynchronize by a few milliseconds, then the timewindow 332 can be set to last 100 milliseconds to counteract thedesynchronization. In this manner, camera server 225 can mitigate theeffects of desynchronization as the camera subsystem 202 is scaled tohandle larger spaces by including more cameras 305 and camera clients220. In the example of FIG. 3J, camera server 225 sets the duration oftime window 1 332A to be between T0 and T1 and the duration of timewindow 2 332B to be between T1 and T2. Camera server 225 can set theduration of the time windows 332 to be any suitable amount to mitigatethe effects of desynchronization. In certain embodiments, T0 may be thetime when the cameras 305 in the camera subsystem 202 have finishedinitializing.

FIG. 3K shows an embodiment where camera server 225 uses cursors 335 toassign frame data 330 to time windows 332. Each cursor 335 maycorrespond to a particular camera client 220 in the camera subsystem202. In the example of FIG. 3K, cursor 335A corresponds to camera client1 220A, cursor 335B corresponds to camera client 3 220C, and cursor 335Ccorresponds to camera client 2 220B. Each cursor 335 points to aparticular time window 332. When frame data 330 is received from acamera client 220, that frame data 330 is generally assigned to the timewindow 332 to which the cursor 335 for that camera client 220 points.For example, if frame data 330 is received from camera client 1 220A,then that frame data 330 is generally assigned to time window 1 332A,because cursor 335A is pointing to time window 1 332A.

Camera server 225 may determine whether to advance cursor 335A whenframe data 330 is received from the camera client 220 corresponding tothat cursor 335. If that frame data 330 has a timestamp 324 that belongsin a subsequent time window 332, then camera server 225 may advance thecursor 335 to that time window 332, thereby indicating that cameraserver 225 is not expecting to receive any more frame data 330 from thatcamera client 220 that belongs in a prior time window 332. In thismanner, camera server 225 can quickly and efficiently assign frame data330 to time windows 332 without checking every time window 332 whenframe data 330 is received. For example, if camera client 2 220B isfaster at sending information than camera client 1 220A and cameraclient 3 220C, then cursor 335C may advance far ahead of cursors 335Aand 335B. When camera server 225 receives frame data 330 from cameraclient 2 220B, camera server 225 need not check every time window 332beginning from time window 1 332A to determine to which time window 332that frame data 330 should be assigned. Rather, camera server 225 canstart at the time window 332 to which cursor 335C points. In otherwords, camera server 225 need not first check whether a timestamp 324 inthe frame data 330 from camera client 2 220B indicates a time that fallswithin time window 1 332A and then whether that time falls within timewindow 2 332B. Instead, camera server 225 can first check whether thattime falls within time window 3 332C and ignore checking whether thattime falls within time window 1 332A and time window 2 332B. As aresult, the frame data 330 is quickly and efficiently assigned to thecorrect time window 332.

FIG. 3L illustrates camera server 225 moving out for processing framedata 330 that has been assigned to particular time windows 332.Generally, camera server 225 may determine that the frame data 330assigned to a particular time window 332 is ready for processing. Inresponse to that determination, camera server 225 may move the framedata 330 from a particular time window 332 to a task queue 336.Information in the task queue 336 is then processed to determine thephysical location of people or objects within a space during particulartime windows 332.

Camera server 225 determines that frame data 330 assigned to aparticular time window 332 is ready for processing in any suitablemanner. For example, camera server 225 may determine that a particulartime window 332 is ready for processing when that time window 332 hasframe data 330 for frames 320 from a sufficient number of cameras 305.Camera server 225 may use a threshold 338 to make this determination.When a particular time window 332 has been assigned frame data 330 forframes 320 from a number of cameras 305 that exceeds threshold 338,camera server 225 may determine that that time window 332 is ready forprocessing and move the information for that time window 332 to the taskqueue 336. For example, assume threshold 338 indicates that frame data330 for frames 320 from ten cameras 305 of an array 300 of twelvecameras 305 need to be received before a time window 332 is ready forprocessing. If time window 332 contains frame data 330 for frames 320from only eight cameras 305, then camera server 225 determines that timewindow 332 is not ready for processing, and as a result, time window 332waits to be assigned frame data 330 for frames 320 from additionalcameras 305. When time window 332 has received frame data 330 for frames320 from ten or more cameras 305, camera server 225 determines that timewindow 332 is ready for processing and moves frame data 330 in timewindow 332 to task queue 336.

Camera server 225 may also determine that a particular time window 332is ready for processing when a subsequent time window 332 has receivedframe data 330 for frames 320 from a number of cameras 305 exceedingthreshold 338. Using the previous example, even if time window 1 332Ahas been assigned frame data 330 for frames 320 from eight cameras,camera server 225 may nevertheless determine that time window 1 332A isready for processing when time window 2 332B has been assigned framedata 330 for frames 320 from ten or more cameras 305 (e.g., from everycamera 305 in camera array 300). In this scenario, camera server 225 mayassume that no additional frame data 330 will be assigned to time window1 332A because frame data 330 for frames 320 from a sufficient number ofcameras 305 has been assigned to a subsequent time window 2 332B. Inresponse, camera server 225 moves frame data 330 in time window 1 332Ato task queue 336.

Camera server 225 may also determine that a particular time window 332is ready for processing when that time window 332 has been awaitingprocessing for a certain period of time. For example, if an error or bugoccurs in the system and frames 320 from a number of cameras 305 are notsent or are lost, then a time window 332 may not receive frame data 330for frames 320 from enough cameras 305. As a result, processing for thattime window 332 may stall or be delayed. Camera server 225 may use atimeout or age-out beyond which a time window 332 does not wait forprocessing. Thus, when the time window 332 has not been processed for acertain period of time exceeding the timeout or the age-out, cameraserver 225 may nevertheless send the frame data 330 in that time window332 to the task queue 336. Using the previous example, assume thetimeout is 200 milliseconds. If time window 1 332A has been stuck withframe data 330 from frames 320 from eight cameras 305 for over 200milliseconds, camera server 225 may determine that time window 1 332Ahas waited long enough for additional frame data 330 and that timewindow 1 332A is ready for processing. In response, camera server 225moves frame data 330 in time window 1 332A to task queue 336.

In certain embodiments, when a time window 332 times out or ages out,camera server 225 may adjust threshold 338 so that future time windows332 are less likely to time out or age out. For example, camera server225 may lower threshold 338 when a time window 332 times out or agesout. Likewise, camera server 225 may increase threshold 338 when asubsequent time window 332 does not time out or age out. Camera server225 may adjust threshold 338 based on the number of cameras 305 thathave sent information for a particular time window 332. For example, ifa particular time window 332 times out or ages out when it has framedata 330 for frames 320 from eight cameras 305, and threshold 338 is tencameras 305, camera server 225 may reduce threshold 338 to a valuecloser to eight cameras. As a result, that time window 332 may then haveframe data 330 for frames 320 from a sufficient number of cameras 305and be moved to task queue 336. When a subsequent time window 332 doesnot time out because it has received frame data 330 for frames 320 fromnine cameras 305, camera server 225 may adjust threshold 338 towardsnine cameras 305. In this manner, camera server 225 may dynamicallyadjust the threshold 338 to prevent bugs, errors, and/or latency fromcausing delays in the camera subsystem 202.

In certain embodiments, camera server 225 processes time windows 332sequentially. In other words, camera server 225 does not process asubsequent time window 332 until a prior time window 332 is ready forprocessing. In the example of FIG. 3L, camera server 225 may not placetime window 2 332B into the task queue 336 until time window 1 332A hasbeen placed into the task queue 336. In this manner, the progression ofevents in a store 100 is evaluated sequentially (e.g., as the eventsunfold), which allows for proper tracking of the position of people inthe store 100. If time windows 332 were not evaluated sequentially, thenit may seem to the tracking system 132 that the event in the store 100progressed in a different and incorrect order.

FIG. 3M illustrates a task queue 336 of camera server 225. As shown inFIG. 3M, the task queue 336 includes frame data 330 from two timewindows 332. At the beginning of the task queue 336 is frame data 330for frames 320A, 320B, and 320C. Following in the task queue 336 isframe data 330 for frames 320D, 320E, and 320F. Camera server 225 mayprocess the entries in the task queue 336 in order. Thus, camera server225 may first process the first entry of the task queue 336 and processthe frame data 330 for frames 320A, 320B, and 320C. Camera server 225processes an entry of a task queue 336 and then moves that entry to aresult queue.

To process an entry of task queue 336, camera server 225 may combine orcluster the coordinates 322 of the same objects detected by the samecameras 320 to calculate combined coordinates 332 for that object. As aresult of this processing, each time window 332 should include only oneset of coordinates 322 per object per camera 305. After this processing,the combined coordinates 322 are placed into a result queue. FIG. 3Nillustrates a result queue 340 of camera server 225. As seen in FIG. 3N,result queue 340 includes the combined coordinates 332 for two timewindows 332.

As an example, camera server 225 first processes the first entry in thetask queue 336, which includes frame data 330 for frames 320A, 320B, and320C. Frames 320A and 320C are from the same camera 320A. As a result,camera server 225 may use the frame data 330A and 330C for frames 320Aand 320C to calculate a combined coordinate 322 for the people orobjects detected by camera 320A. As seen in FIG. 3N, camera server 225has determined combined coordinates 322 (x₁₃, y₁₃), and (x₁₄, y₁₄) and acombined height z₇ for object 1 detected by camera 1 305A and combinedcoordinates 322 (x₁₅, y₁₅) and (x₁₆, y₁₆) and a combined height z₈ forobject 2 detected by camera 1 305A. These combined coordinates 322 andcombined heights are the combined coordinates 322 and combined heightsfor the man and the woman in the video frames 302 received by camera305A during the first time window 332A. Likewise, camera server 225 maydetermine combined coordinates 322 and combined heights for the objectsdetected by camera 2 305B during the first time window 332A. Forexample, camera server 225 may use frame data 330B for frame 320B (andframe data 330 for any other frames 320 received by camera 2 305B duringthe first time window 332A) to determine combined coordinates 322 (x₁₃,y₁₃), and (x₁₄, y₁₄) and a combined height z₇ for object 1 detected bycamera 2 305B and combined coordinates 322 (x₁₅, y₁₅) and (x₁₆, y₁₆) anda combined height z₈ for object 2 detected by camera 2 305B. Cameraserver 225 may determine combined coordinates 322 for each objectdetected by cameras 305 in the first time window 332A in this manner.

Camera server 225 then determines combined coordinates 322 for objectsdetected by the cameras 305 during the second time window 332B in asimilar fashion. For example, camera server 225 may use frame data 330Efor frame 320E (and frame data 330 for any other frames 320 received bycamera 1 305A during the second time window 332B) to determine combinedcoordinates 322 (x₁₇, y₁₇), and (x₁₈, y₁₈) and a combined height z₉ forobject 1 detected by camera 1 305A and combined coordinates 322 (x₁₉,y₁₉) and (x₂₀, y₂₀) and a combined height z₁₀ for object 2 detected bycamera 1 305A. Camera server 225 may also use frame data 330D and 330Ffor frames 320D and 320F to determine combined coordinates 322 (x₁₇,y₁₇), and (x₁₈, y₁₈) and a combined height z₉ for object 1 detected bycamera 2 305B and combined coordinates 322 (x₁₉, y₁₉) and (x₂₀, y₂₀) anda combined height z₁₀ for object 2 detected by camera 2 305B.

Camera server 225 calculates combined coordinates 322 and combinedheights in any suitable manner. For example, camera server 225 maycalculate combined coordinates 322 and combined heights by taking theaverage of the coordinates 322 and the heights of particular objectsdetected by the same camera 305 in a particular time window 332. Usingthe example in FIG. 3M, camera server 225 may calculate combinedcoordinates 322 (x₁₃, y₁₃) for camera 1 305A by taking the average ofcoordinates 322 (x₁, y₁) and (x₅, y₅) from frame data 330A and 330C.Similarly, camera server 225 may determine the combined coordinate 322(x₁₄, y₁₄) for camera 1 305A by taking the average of coordinates 322(x₂, y₂) and (x₆, y₆) from frame data 330A and 330C. Camera server 225may determine combined height z₇ for camera 1 305A by taking the averageof heights z₁ and z₃ from frame data 330A and 330C. Similarly, cameraserver 225 may determine combined coordinates 322 (x₁₇, y₁₇) for camera2 305B by taking the average of coordinates 322 (x₅, y₅) and (x₉, y₉)from frame data 330D and 330F. Likewise, camera server 225 may determinecombined coordinates 322 (x₁₈, y₁₈) for camera 2 305B by taking theaverage of coordinates 322 (x₆, y₆) and (x₁₀, y₁₀) from frame data 330Dand 330F. Camera server 225 may determine combined height z₉ for camera2 305B by taking the averages of heights z₃ and z₅ from frame data 330Dand 330F. Camera server 225 takes these averages because these are thecoordinates 322 and heights for the same object determined by the samecamera 305 during the same time window 332.

Camera server 225 may follow a similar process to determine or tocalculate the combined coordinates for object 2 detected by cameras 1305A and 2 305B. Camera server 225 may calculate combined coordinates322 (x₁₅, y₁₅) for camera 1 305A by taking the average of coordinates322 (x₃, y₃) and (x₇, y₇) from frame data 330A and 330C. Similarly,camera server 225 may determine the combined coordinate 322 (x₁₆, y₁₆)for camera 1 305A by taking the average of coordinates 322 (x₄, y₄) and(x₈, y₈) from frame data 330A and 330C. Camera server 225 may determinecombined height z₈ for camera 1 305A by taking the average of heights z₂and z₄ from frame data 330A and 330C. Similarly, camera server 225 maydetermine combined coordinates 322 (x₁₉, y₁₉) for camera 2 305B bytaking the average of coordinates 322 (x₇, y₇) and (x₁₁, y₁₁) from framedata 330D and 330F. Likewise, camera server 225 may determine combinedcoordinates 322 (x₂₀, y₂₀) for camera 2 305B by taking the average ofcoordinates 322 (x₈, y₈) and (x₁₂, y₁₂) from frame data 330D and 330F.Camera server 225 may determine combined height z₁₀ for camera 2 305B bytaking the averages of heights z₄ and z₆ from frame data 330D and 330F.

Camera server 225 uses any other suitable calculation to calculatecombined coordinates and combined heights. For example, camera server225 may take a median of coordinates 322 and heights for objectsdetected by the same camera 305 during a time window 332. Camera server225 may also use clustering processes to calculate the combinedcoordinates 322 and combined heights. For example, camera server 225 mayuse K-means clustering, Density-based spatial clustering of applicationswith noise (DBSCAN), k-medoids, gaussian mixture models, andhierarchical clustering to calculate combined coordinates 322 andcombined heights.

After camera server 225 has calculated the combined coordinates 322 andcombined heights, camera server 225 has determined the coordinates 322for each object detected by each camera 305 during a time window 332.However, camera server 225 may perform additional processing todetermine whether the object detected by different cameras 305 are thesame object. Camera server 225 may use linking and homography todetermine which objects detected by which cameras 305 are actually thesame person or object in a space. Camera server 225 may then take thecombined coordinates 322 for those objects from the different cameras305 and employ homography to determine a physical location for thatperson or object in the physical space during a time window 332.Embodiments of this process are described in U.S. patent applicationSer. No. 16/663,710 entitled, “Topview Object Tracking Using a SensorArray” (attorney docket no. 090278.0180), the contents of which areincorporated by reference herein in its entirety. In this manner, cameraserver 225 determines the physical locations of people and/or objectswithin the space during particular time windows 332.

In particular embodiments, camera clients 220 may also use the same timewindows 332 as camera server 225 to communicate frame data 330 inbatches to camera server 225. As seen in FIG. 3O, camera client 220assigns frame date 330 to time windows 332 based on the timestamps 324within that frame data 330. Camera client 220 may determine that aparticular time window 332 is ready to be communicated to camera server225 in a similar way as camera server 225 determines a time window 332is ready for processing. When camera client 220 determines that aparticular time window 332 is ready (e.g., when each camera 305communicatively coupled to camera client 220 has communicated a frame inthat time window 332), camera client 220 communicates the frame data 330assigned to that time window 332 as a batch to the camera server 225. Inthis manner, camera server 225 may assign frame data 330 to time windows332 even more quickly and more efficiently because camera server 225receives the frame data 330 for a time window 332 as a batch from cameraclient 220.

In certain embodiments, even if camera server 225 and camera clients 220are not synchronized, camera server 225 can account fordesynchronization that occurs (e.g., by desynchronized internal clocks302, by latency differences between camera clients 220 to camera server225, by processing speed differences between camera clients 220, etc.)by adjusting the timestamps 324 in frame data 330. FIG. 3P shows cameraserver 225 adjusting timestamps 324. As discussed previously, frame data330 includes a timestamp 324 generated by camera client 220 thatindicates when camera client 220 received a frame 320. In the example ofFIG. 3P, frame data 330 indicates that camera client 220 received frame320 at time t₁. If the camera clients 220 and camera server 225 are notsynchronized, then the timestamp 324 t₁ is relatively meaningless tocamera server 225 because the camera server 225 cannot be assured thattimestamps 324 from different camera clients 220 are accurate relativeto each other. Thus, it is difficult, if not impossible, to preciselyanalyze frame data 330 from different and/or multiple camera clients220.

Camera server 225 can adjust timestamps 324 for particular cameras 305to account for desynchronization. Generally, camera server 225determines a delay for each camera 305 by tracking the delay for priorframes 320 from that camera 305. Camera server 225 then adjuststimestamps 324 for frame data 330 for frames 320 from that camera 305 bythe determined delay. In the example of FIG. 3P, camera server 225determines a delay for camera 1 305A by determining, for each frame 320(x) from camera 1, the difference in time (labeled Δ_(x)) between thetimestamp 324 indicated in frame data 330 for that frame 320 (labeledt_(x)) and the time camera server 225 received the frame data 330(labeled T_(x)). Camera server 225 calculates an average delay (labeledΔ) by averaging the differences in time (Δ_(x)) for a prior number offrames 320. In the example of FIG. 3P, camera server 225 averages thedifferences in time for the previous thirty frames 320 to determine theaverage delay. Camera server then adds the average delay (A) to thetimestamp 324 for the frame data 330 to adjust the timestamp 324 toaccount for desynchronization. In this manner, camera server 225 andtracking system 132 can function properly even if camera clients 220 andcamera server 225 are not synchronized (e.g., according to a clocksynchronization protocol).

5. Example Method

FIGS. 3Q and 3R are flowcharts illustrating an example method 342 ofoperating the camera subsystem 202. In particular embodiments, variouscomponents of the camera subsystem 202 perform the steps of method 342.Generally, by performing method 342, the camera subsystem 202 determinesthe physical position of people or objects within a space.

As seen in FIG. 3Q, method 342A begins with cameras 305A and 305Bgenerating and communicating frames 320A and 320D to camera clients 220Aand 220B, respectively. Camera clients 220A and 220B then determinecoordinates 322 for two people detected in frames 320A and 320B. Thesecoordinates may define bounding areas 325 around these people.

Camera 305A then generates frame 320B and communicates frame 320B tocamera client 220A. Camera client 220A generates coordinates 322 for twopeople shown in frame 320B. During that process, camera 305B generatesframe 320E and communicates frame 320E to camera client 220B. Cameraclient 220B then determines coordinates 322 for two people detected inframe 320E. Camera 305A then generates frame 320C and communicates frame320C to camera client 220A. Camera client 220A determines coordinates322 for two people detected in frame 320C. Importantly, FIG. 3Q showsthat frames from cameras 305A and 305B may not be generated andcommunicated simultaneously or synchronously. Additionally, coordinatesfor people detected in frames 320 may not be generated simultaneously orsynchronously in camera clients 220A and 220B.

FIG. 3R shows method 342B which continues from method 342A of FIG. 3Q.As seen in FIG. 3R, camera client 220A generates frame data 330 from thecoordinates 322 for the two people detected in frame 320A. Likewise,camera client 220B generates frame data 330 using the coordinates 322for the two people detected in frame 320D. Camera clients 220A and 220Bcommunicate the frame data 330 to camera server 225. Camera client 220Agenerates additional frame data 330 using the coordinates 322 for thetwo people detected in frame 320B. Camera client 220A then communicatesthat frame data 330 to camera server 225. Camera server 225 may assignthe frame data 330 to a time window 332. Camera server 225 may determinethat that time window 332 is ready for processing in step 344 and, inresponse, place the frame data 330 in that time window 332 into a taskqueue 336 in step 346. Camera server 225 may then combine or cluster thecoordinates 322 in that time window 322 to determine combinedcoordinates 322 in step 348. For example, camera server 225 may averagethe coordinates 322 in that time window to determine combinedcoordinates 322 for the people detected by the different cameras 305during that time window 332. Camera server 225 may then map the peopledetected by the different cameras 305 to people in the space in step350. Camera server 225 may then determine the positions of the peopleduring that time window 332 in step 352. Camera server 225 communicatesthese determined positions to central server 240.

Modifications, additions, or omissions may be made to method 342depicted in FIGS. 3Q and 3R. Method 342 may include more, fewer, orother steps. For example, steps may be performed in parallel or in anysuitable order. While discussed as particular components of camerasubsystem 202 performing the steps, any suitable component of camerasubsystem 202 may perform one or more steps of the method.

6. Other Features

In particular embodiments, the camera subsystem 202 may include a secondcamera array that operates in tandem with the first camera array 300 ofthe camera subsystem 202. FIG. 3S shows an embodiment that includes twocamera arrays 300 and 354. Camera array 300 includes cameras 305M.Camera array 354 includes cameras 305N. Cameras 305N operate in the sameway as cameras 305M and can be used to determine positions of objectsand/or people in a space using the same techniques described using FIGS.3A-3R.

Each camera 305N is positioned slightly offset from a camera 305M ofcamera array 300. In this manner, cameras 305M capture video that issimilar to the video captured by cameras 305N. In certain embodiments,cameras 305M may use different versions of software or differentversions of software may be used to process video from cameras 305Mrelative to cameras 305N. In this manner, newer software can be run forcameras 305N to test the effectiveness of that software. The testing ofthat software does not interrupt the operation of the camera subsystem202 because cameras 305M may still be using the previous software, whichalso acts as a baseline for comparing against the operation of the newsoftware running on cameras 305N. For example, the accuracy of theposition tracking provided by the new software can be determined andcompared against the accuracy provided by the old software. If the newsoftware is less accurate than the old software, then the old softwareshould continue to be used.

In certain embodiments, camera server 225 can retrieve video footagefrom camera clients 220 or a shared memory if the camera server 225 isunable to determine the positions of people based on the frame data 330from the camera clients 220. FIG. 3T shows a camera server 225retrieving videos 302 from camera clients 220 and/or shared memory 356.Generally, camera clients 220 store video received from cameras locallyor in a shared memory 356. That video 302 is then made available tocamera server 225 if camera server 225 cannot determine the positions ofpeople based on frame data 330. Camera server 225 may analyze video 302to determine the positions of people in the space. Camera server 225 mayperform better and more accurate analysis of the raw video footage thancamera clients 220, and thus, camera server 225 may generate moreaccurate frame data 330 than camera clients 220. In some embodiments,camera server 225 may have frame data 330 from one camera client 220that conflicts or does not align with frame data 330 from another cameraclient 220. Camera server 225 can retrieve the raw video footage todetermine which frame data 330 should be accepted and used.

In the example of FIG. 3T, camera client 220A stores video 302A locallyor in shared memory 356. Camera client 220B stores video 302B locally orin shared memory 356. When camera server 225 is unable to determine thepositions of people based on frame data 330, camera server 225 sends arequest 358 to camera client 220A and/or shared memory 356. In response,camera client 220A and/or shared memory 356 send video 302A to cameraserver 225. Camera server 225 may then analyze the video 302A todetermine the positions of people in the space.

FIG. 3U illustrates an embodiment of a block diagram of an examplecamera subsystem 242 from FIG. 2B. As seen in FIG. 3U, each camera 305is operably coupled with a corresponding camera client 220. In theexample of FIG. 3U, each of cameras 305 a-305 i is operably coupled withits corresponding camera client 220 a-220 i. Generally, cameras 305 andcamera clients 220 are positioned above a space so that cameras 305 cancapture overhead videos of portions of the space, and communicate thosevideos to their corresponding camera clients 220. Although in theillustrated example, the camera array 300 is shown with twelve cameras305 operably coupled with a corresponding twelve camera clients 220, thecamera subsystem 242 may include any number and combination of cameras305 operably coupled with corresponding camera clients 220. Thus, camerasubsystem 242 contemplates a one-to-one relationship between cameras 305and camera clients 220, a one-to-many relationship between cameras 305and camera clients 220, or a many-to-one relationship between cameras305 and camera clients 220.

In one embodiment, in the camera subsystem 242, each camera client 220may be positioned adjacent to its corresponding camera 305. With thismethod, the videos can be more easily synchronized due to lesstransmission time of the videos from cameras 305 to camera clients 220.Cameras 305 are communicatively coupled to their corresponding cameraclients 220 using any suitable medium. For example, cameras 305 may behardwired to camera clients 220. As another example, cameras 305 maywirelessly couple to camera clients 220 using any suitable wirelessprotocol (e.g., WiFi). Cameras 305 communicate captured videos throughthe communication medium to the camera clients 220.

The camera clients 220 are communicatively coupled to the clusterservers 248. The camera clients 220 are configured to process the videos(received from the cameras 305), generate, and communicate frames 314and tracks 316 to the cluster servers 248. The operation of generatingframes 314 and tracks 316 is described in the corresponding descriptionof FIG. 3W. The cluster servers 248 store and maintain the frames 314and tracks 316 using one or more labels or indexes associated with theframes and tracks 316. The cluster servers 248 store the frames 314 andtracks 316 such that they are retrievable using one or more of theircorresponding labels. In one embodiment, the cluster servers 248 maystore the frames 314 and tracks 316 separately from each other, forexample, in different servers. In one embodiment, the cluster servers248 may store the frames 314 and tracks 316 in the same server. Thecluster servers 248 communicate a particular set of frames 314 to thecamera server 225 upon receiving a request from the camera server 225 tosend the particular set of frames 314 using one or more labels of theparticular set of frames 314. Similarly, the cluster servers 248communicate a particular set of tracks 316 to the camera server 225 uponreceiving a request from the camera server 225 to send the particularset of tracks 316 using one or more labels of the particular set oftracks 316.

FIG. 3V illustrates an initialization of the camera subsystem 242.Generally, during the initialization, each camera 305 of the cameraarray 300 starts up and begins sending videos 302 to its correspondingcamera client 220 from the camera client array 222. Additionally, cameraclients 220 and camera server 225 synchronize internal clocks 304. Afterthe cameras 305 in camera array 300 have started up and after theinternal clocks 304 are synchronized, camera clients 220 may beginprocessing videos 302 and communicating data 312 to cluster servers 248to store and maintain the received data 312. For example, data 312 mayinclude frames 314. In another example, data 312 may include tracks 316.The cluster servers 248 may then forward the data 312 to the cameraserver 225 upon receiving a request to receive those data 312 to performthe tracking operations of the camera subsystem 202.

During the initialization, the camera clients 220 and the camera server225 synchronize their internal clocks 30, similar to that described inFIG. 3F. The cameras 305 may power on and perform a startup sequencesimilar to that described in FIG. 3F. As seen in FIG. 3V, the firstcamera 305 a sends videos 302 a and 302 b to the camera client 220 a.Similarly, the second camera 305 b sends videos 302 c and 302 d to thecamera client 220 b. Since in the illustrated embodiment, the cameraclients 220 are migrated to be adjacent to the cameras 305, videos 302a-d travel a short distance to arrive at the camera clients 220. Thus,videos 302 a-d are more easily synchronized to the actual time ofcapturing at the cameras 305. This reduces the synchronizationcomplexity which, in turn, reduces the synchronization computation time,and provides a more seamless tracking process.

FIG. 3W illustrates an operational flow of generating frames 314 andtracks 316. In particular, FIG. 3W illustrates an operational flow ofprocessing the frames 320 of videos 302 at the camera client 220,generating frames 314 from the color frames 364 and depth frames 366,communicating the frames 314 to a server (e.g., a first server) serverfrom the cluster servers 248, generating tracks 316 from the depthframes 366, and communicating the tracks 316 to a server (e.g., a secondserver) from the cluster servers 248.

As seen in FIG. 3X, the operational flow begins when the camera 305sends the frames 320 to the camera client 220 for processing. The camera305 may routinely send a plurality of frames 320 to the camera client220 according to the speed of the camera 305. For example, the camera305 may send thirty Frames Per Second (FPS) or any other number offrames per second to the camera client 220. The frames 320 include colorframes 364 and depth frames 366. Color frames 364 represent orcorrespond to visual colors of objects in the frames 320. Depth frames366 represent or correspond to distances of objects in the frames 320from the camera 305. For example, the depth frames 366 may representthree-dimensional point clouds corresponding to distances of objects inthe frames 320 from the camera 305. The camera client 220 labels orindexes each corresponding color frame 364 and depth frame 366 with atimestamp 368 when they arrive at the camera client 220. The cameraclient 220 also labels or indexes the color frames 364 and depth frames366 with a camera identifier 370 associated with the camera 305. Thecamera identifier 370 may be related to a position of the camera 305 inthe store 100, for example, a region or a zone that is in thefield-of-view of the camera 305. During the processing of color frames364 and depth frames 366, the camera client 220 generates frames 314 andtracks 316.

Generating Frames 314

In one embodiment, for generating the frames 314, the camera client 220may perform a compression step 372 on the color frames 364. Similarly,the camera client 220 may perform a compression step 372 on the depthframes 366. For example, the camera client 220 may individually compresseach color frame 364 from the color frames 364. Similarly, the cameraclient 220 may individually compress each depth frame 366 from the depthframes 366. In other examples, the camera client 220 may compress anynumber of color frames 364 together. Similarly, the camera client 220may compress any number of depth frames 366 together. In compressing thecolor frames 364, the camera client 220 may use any suitable compressiontechnique. In one example, the camera client 220 may use a lossycompression where the size of the color frames 364 is reduced. Inanother example, the camera client 220 may use a lossless compressionwhere the size of the color frames 364 remains the same. Similarly, incompressing the depth frames 366, the camera client 220 may use anycompression technique, such as a lossy compression, a losslesscompression, etc. In one embodiment, the camera client 220 may notcompress one or more of the color frames 364 and depth frames 366.

The camera client 220 then performs a serialization step 374 on thecolor frames 364 and depth frames 366. In the serialization step 374,the color frames 364 and depth frames 366 are structured into a formatthat can be transmitted and reconstructed later. This allows forcommunicating the frames 314 between the camera client 220, the clusterservers 248, the camera server 225, and central server 240. In anexample, the camera client 220 may serialize the color frames 364 anddepth frames 366 using a Protocol Buffer (ProtoBuf) serializationtechnique. In this process, the color frames and depth frames may bestructured or formatted into JavaScript Object Notation (JSON)dictionaries. In other examples, the camera client 220 may use any otherserialization techniques.

After the serialization step 374, the camera client 220 sends the frames314 to a first server of the cluster servers 248. The frames 314 includethe color frames 364 and depth frames 366 labeled with theircorresponding timestamps 368 and the camera identifier 370. With thismethod, each of the frames 314 can be referenced or queried using one ormore of its corresponding labels, e.g., timestamps 368 and the cameraidentifier 370. Thus, the cluster servers 248 can communicate aparticular set of frames 314 to the camera server 225 when the cameraserver 225 sends a request to the cluster servers 248 to communicate theparticular set of frames 314 using one or more labels associated withthe particular set of frames 314. In a particular example, the cameraserver 225 may send a request to the cluster servers 248 to send frames314 from a first timestamp 368 to a second timestamp 368 (i.e., in aparticular timestamp range) from the camera 305 with a particularidentifier number 370. The camera server 225 may performde-serialization to reconstruct the frames 314 for processing. Thecamera client 220 may routinely send the frames 314 to the first serverfrom the cluster servers 248, for example, every second, every fewseconds, or any other duration.

Generating Tracks 316

In one embodiment, the camera client 220 may asynchronously generate thetracks 316 with respect to generating the frames 314 discussed above. Inthis process, the camera client 220 performs a contour detection step376 on each depth frame 366 and a tracking identification associationstep 390 for people detected in each depth frame 366. The contourdetection step 376 and tracking identification association step 390 aredescribed in detail in FIGS. 8-15. The corresponding description belowincludes a brief description of the contour detection step 376 andtracking identification association step 390. The contour detection step376 begins when the camera client 220 detects one or more contours 378associated with a person in a first depth frame 366 (assuming that thefirst depth frame 366 is showing the person). In one embodiment, thecamera client 220 may detect a plurality of contours 378 associated withthe person at different heights or depths corresponding to differentdistances from the camera 305. For example, the camera client 220 mayclip the first depth frame 366 into different segments where eachsegment represents a different distance from the camera 305. In theillustrated example in FIG. 3W, the camera client 220 detects a contour378 a at a low depth 360 a, a contour 378 b at a medium depth 360 b, anda contour 378 c at a high depth 360 c. In some embodiments, the cameraclient 220 may clip the depth frame 366 into any number of segments anddetect any number of contours 378 associated with the person in thefirst depth frame 366. Each contour 378 is represented by pixelcoordinates around the person. For each contour 378 associated with theperson, the camera client 220 detects a bounding area 382 around theperson in the first depth frame 366. The camera client 220 detects thebounding area 382 based on the pixel coordinates of the contour 378. Seethe corresponding descriptions of FIGS. 19-21 for further description ofpixel coordinates of the contour 378.

In some embodiments, the camera client 220 generates a plurality ofbounding areas 382 based on the pixel coordinates of each contour 378.For example, the camera client 220 generates the plurality of boundingareas 382 based on pixel coordinates of each contour 378 using acontour-based detection technique described in FIGS. 8-18. In theillustrated example in FIG. 3W, the camera client 220 may detect aplurality of bounding areas 382 a for the contour 378 a, a plurality ofbounding areas 382 b for the contour 378 b, and a plurality of boundingareas 382 c for the contour 378 c. The camera client 220 then performs anon-maximum suppression 384 on each of the plurality for bounding areas382 a, 328 b, and 382 c to suppress one or more bounding areas 382 thatare overlapped with other bounding areas 382 below a threshold area. Seethe corresponding description of the non-maximum suppression 384 inFIGS. 11 and 12.

In brief, for example, the camera client 220 calculates a score for eachbounding area 382. The score, for example, may represent an extent towhich that bounding area 382 is similar to or correspond to the otherbounding areas 382. For example, in the cases of performing thenon-maximum suppression 384 on the plurality of bounding areas 382 a,assume that the camera client 220 is determining whether a firstbounding area 382 from the plurality of bounding areas 382 a is similarto a second bounding area 382 from the plurality of bounding areas 382a. In this process, 382 may use one or more metrics including: 1) anoverlapping region between the first bounding area 382 and the secondbounding area 382, 2) a ratio of intersection over union region betweenthe first bounding area 382 and the second bounding area 382, and 3) adistance between the center of the first bounding area 382 and thecenter of the second bounding area 382. If the camera client 220determines that 1) the overlapping region between the first boundingarea 382 and the second bounding area 382 is above a threshold region,2) the ratio of intersection over union region between the firstbounding area 382 and the second bounding area 382 is above a thresholdvalue, and 3) the distance between the center of the first bounding area382 and the center of the second bounding area 382 is below a thresholddistance, the camera client 220 determines that the second bounding area382 is similar to or correspond to the first bounding area 382. Thecamera client 220 may use these metrics to calculate a score for eachbounding area 382.

Thus, the camera client 220 may identify a subset of the bounding areas382 with a score that is greater than a threshold value (e.g., 80%), anddetermine a region based on this identified subset. For example, theregion may be the bounding area 382 with the highest score or a combinedbounding areas 382 comprising regions shared by bounding areas 382 witha score that is above the threshold value. The camera client 220, basedon the bounding areas 382, detects coordinates or pixel coordinates ofthe person in the first depth frame 366. The pixel coordinates of theperson generally refer to the location of the person (i.e., row andcolumn) in the depth frame 366. Likewise, the camera client 220 mayperform the non-maximum suppression 384 on the plurality of boundingareas 382 b associated with contours 378 b, and the plurality ofbounding areas 382 c associated with contours 378 c.

After performing the non-maximum suppression 384, the camera client 220associates a first tracking identification 386 to the person in thetracking identification association step 390, where the first trackingidentification 386 is linked to historical detections 388 associatedwith the person. The historical detections 388 associated with theperson may include one or more of the bounding areas 382 detected atdifferent depths 380, contours 378, segmentation masks, etc. The cameraclient 220 determines the first tracking identification 386 for theperson that is the best match based on the historical detections 388associated with the person. For example, the historical detections 388associated with the person may include one or more of bounding areas382, contours 378, and segmentation masks in previous depth frames 366that show that person. The first tracking identification 386 may bereferred to as a local tracking identification 386 associated with theperson in the first depth frame 366. The first local trackingidentification 386 may serve as a hint to the camera server 225 todetermine a global tracking identification 386 for the person in thestore 100. In other words, the camera server 225 may or may not use thefirst local tracking identification 386 to determine the global trackingidentification 386 for the person.

The camera client 220 then performs a serialization step 372 on theidentified bounding areas 382, historical detections 388, and the firsttracking identification 386. For example, in the serialization step 372,the metadata describing the identified bounding areas 382, historicaldetections 388, and the first tracking identifications 386 arestructured into a format that can be transmitted and reconstructedlater. This process may be similar to that described above with respectto the serialization step 372 in generating the frames 314. The cameraclient 220 then sends the tracks 316 to a second server from the clusterservers 248.

In one embodiment, the frames 314 and tracks 316 may have a one-to-onecorrelation to each other. As such, if a track 316 generated from aparticular depth frame 366 is empty (meaning no person was detected inthat depth frame 366), the empty track 316 may be sent to the clusterservers 248 so that a record of the empty track 316 is recorded at thecluster servers 248.

The tracks 316 include metadata describing the historical detections 388associated with the person and the tracking identifications 386. Thetracks 316 are labeled with their corresponding timestamps 368, thecamera identifier number 370, historical detections 388 (e.g., boundingareas 382, contours 378, segmentation masks, etc.), trackingidentifications 386, among others. Thus, each of the tracks 316 can bereferenced or queried using one or more of its corresponding labels.

Thus, the cluster servers 248 can communicate a particular set of tracks316 to the camera server 225 when the camera server 225 sends a requestto the cluster servers 248 to communicate the particular set of tracks316 using one or more labels associated with the particular set oftracks 316. The camera server 225 may perform de-serialization toreconstruct the tracks 316 for processing. In a particular example, thecamera server 225 may send a request to the cluster servers 248 to sendtracks 316 from a first timestamp 368 to a second timestamp 368 (i.e., atimestamp range) from the camera 305 with a particular identifier number370.

Similarly, the camera client 220 can determine a second trackingidentification 386 for the person in a second depth frame 366 based onhistorical detections 388 associated with the person. The camera server225 can determine the global tracking identification 386 associated withthe person from the first and second tracking identifications 386. Forexample, the camera server 225 can determine the global trackingidentification 386 for the person by comparing historical detections 388(e.g., bounding areas 382) associated with those people in the first andsecond tracks 316. For example, if bounding areas 382 of a person in thefirst and second tracks 316 are similar or correspond to each other, thecamera server 265 determines that the global tracking identification 386for that person is the same as the first or second local trackingidentification 386.

The camera server 236 determines that the bounding areas 382 of theperson in the first and second tracks 316 are similar to each otherusing one or more metrics including: 1) an overlapping region betweenthe bounding areas 382 in the first and second tracks 316, 2) a ratio ofintersection over union region between the bounding areas 382 in thefirst and second tracks 316, and 3) a distance between the centers ofthe bounding areas 382 in the first and second tracks 316. If the cameraclient 220 determines that 1) the overlapping region bounding areas 382is above a threshold region, 2) the ratio of intersection over unionregion between bounding areas 382 is above a threshold value, and 3) thedistance between the centers of the bounding areas 382 is below athreshold distance, the camera client 220 determines that the boundingareas 382 associated with the person in the first and second tracks 316are similar to each other. The camera server 225 then can determine aglobal position of the person in the store 100 using the coordinates ofthe historical detections 388 (e.g., bounding areas 382) associated withthe person.

7. Example Method

FIGS. 3X and 3Y illustrate flowcharts of an example method 342 ofoperating the camera subsystem 242. In particular embodiments, variouscomponents of the camera subsystem 242 perform the steps of method 342.Generally, by performing method 342, the camera subsystem 202 determinesthe physical positions of people or objects within a space. FIG. 3Xillustrates a first portion of the method 342 (i.e., method 342 a); andFIG. 3Y illustrates a second portion of the method 342 (i.e., method 342b).

As seen in FIG. 3X, method 342 a begins with camera 305 a generating andcommunicating frames 320 a, 320 c, 320 e to camera client 220 a, andcamera 305 b generating and communicating frames 320 b and 320 d tocamera client 220 b. The frames 320 a, 320 c, 320 e may be generated andsequentially communicated to camera client 220 a, and frames 320 b and320 d may be generated and sequentially communicated to camera client220 b. The frames 320 a-320 e may include color frames 364 and depthframes 366 as described in FIG. 3W. From each of frames 320 a, 320 c,320 e, the camera client 220 a detects coordinates 322 for two peopleshown in depth frames 366 related to the frames 320 a, 320 c, 320 e.Similarly, camera client 220 b detects coordinates 322 for two peopleshown depth frames 366 related to the frames 320 b and 320 d. Thesecoordinates 322 may define bounding areas 382 around those people. Asfurther seen in FIG. 3X, the frames 320 a-320 e from cameras 305 a and305 b may not necessarily be generated and communicated simultaneouslyor synchronously. Additionally, coordinates 322 for people detected inframes 320 a-320 e may not necessarily be generated simultaneously orsynchronously in camera clients 220 a and 220 b.

FIG. 3Y illustrates an example flowchart for the method 342 b whichcontinues from method 342 a of FIG. 3X. As seen in FIG. 3Y, cameraclient 220 a generates a frame 314 a from color frame 364 and depthframe 366 associated with frame 320 a, similar to that described in FIG.3W. Camera client 220 a communicates frame 314 a to a server (e.g., afirst server) from the cluster servers 248.

The camera client 220 a also generates a track 316 a from the depthframe 366 associated with frame 320 a, similar to that described in FIG.3W. Camera client 220 a communicates track 316 a to a server (e.g., asecond server) from the cluster servers 248. Likewise, camera client 220b generates a frame 314 b from color frame 364 and depth frame 366associated with frame 320 b, and a track 316 b from the depth frame 366associated with frame 320 b, similar to that described in FIG. 3W.Camera client 220 b communicates the frame 314 b to a server (e.g., thefirst server) from the cluster servers 248. Asynchronously, cameraclient 220 b communicates the track 316 b to a server (e.g., the secondserver) from the cluster servers 248.

Camera client 220 a generates additional frame 314 c and track 316 cfrom the color frame 364 and depth frame 366 associated with frame 320c, and communicates the frame 314 c and track 316 c to the first andsecond server from the cluster servers 248, respectively. Likewise,camera client 220 b generates additional frame 314 d and track 316 dfrom the color frame 364 and depth frame 366 associated with frame 320d, and communicates the frame 314 d and track 316 d to the first andsecond server from the cluster servers 248, respectively.

Cluster servers 248 store and maintain the frames 314 such that each ofthe frames 314 is retrievable using one or more of its correspondinglabels similar to that described in FIG. 3W. Similarly, the clusterservers 248 store and maintain the tracks 316 such that each of thetracks 316 is retrievable using one or more of its corresponding labelssimilar to that described in FIG. 3W.

Camera server 225 sends a request to the cluster servers 248 to receivethe frames 314 a-314 d using one or more of their corresponding labels,e.g., a timestamp range 368, a camera identifier 370, etc., similar tothat described in FIG. 3W. For example, the camera server 225 may send arequest to the cluster servers 248 to receive frames 314 from cameraswith particular camera identifiers 370 (e.g., cameras 305 a and 305 b)within a timestamp range 368.

Camera server 225 may also send a request to the cluster servers 248 toreceive the tracks 316 a-316 d using one or more of their correspondinglabels, e.g., a timestamp range 368, a camera identifier 370, historicaldetections 388 (e.g., bounding areas, contours, segmentation masks,etc.), tracking identifications 386, etc., similar to that described inFIG. 3W. For example, the camera server 225 may send a request to thecluster servers 248 to receive tracks 316 from cameras with particularcamera identifiers 370 (e.g., cameras 305 a and 305 b) within atimestamp range 368.

As such, the camera server 225 may send separate requests to receive theframes 314 a-314 d and tracks 316 a-316 d. In one embodiment, the cameraserver 225 may send a request to receive 1) the frames 314 a-314 d usingone or more of their corresponding labels associated with the frames 314a-314 d and 2) the tracks 316 a-316 d using one or more of theircorresponding labels associated with the tracks 316 a-316 d. When thecamera server 225 receives the frames 314 and tracks 316, it may assignthe frames 314 and tracks 316 to a first time window 332 similar to thatdescribed in FIGS. 3J-3P.

Since the frames 314 can be referenced using one or more of theircorresponding labels, the camera server 225 can query the clusterservers 248 to return a particular frame 314 that was generated in thefirst time window 332 but was not received by the processing time of thefirst time window 332. Similarly, since the tracks 316 can be referencedusing one or more of their corresponding labels, the camera server 225can query the cluster servers 248 to return a particular track 316 thatwas generated in the first time window 332 but was not received by theprocessing time of the first time window 332. For example, assume thattrack 316 d was not sent to the cluster servers 248 in time forprocessing the first time window 332, thus when the camera server 225started processing the first time window 332, the track 316 d was notavailable for processing at the camera server 225. However, since thetrack 316 d was timestamped within the first time window 332, the cameraserver 225 can reference back in time and process the track 316 d in thefirst time window 332. The same applies if any track 316 and/or frame314 were/was not received at the cluster servers 248 in time forprocessing the first time window 332.

Camera server 225 may determine that the first time window 332 is readyfor processing in step 344 and, in response, place the frames 314 andtracks 316 in the first time window 332 into a task queue 336 in step346, similar to that described in FIG. 3R. Camera server 225 may thencombine or cluster the coordinates 322 in the first time window 332 todetermine combined coordinates 322 in step 348. For example, cameraserver 225 may average the coordinates 322 in the first time window 332to determine combined coordinates 322 for the people detected by thedifferent cameras 305 during the first time window 332. Camera server225 may then map the people detected by the different cameras 305 topeople in the space in step 350. Camera server 225 may then determinethe positions of the people during the first time window 332 in step352.

In one embodiment, a position of a person during the first time window332 is determined based on point clouds representing a three-dimensionalspace. For example, the point clouds representing the three-dimensionalspace may be generated by at least one camera client 220 based on acoordinate 322 of the person detected in at least one depth frame 366.In another example, the point clouds representing the three-dimensionalspace may be generated by at least one radio detection and ranging(RADAR) sensor based on a coordinate 322 of the person detected by theRADAR sensor. In another example, the point clouds representing thethree-dimensional space may be generated by at least one LiDAR sensor210 based on a coordinate 322 of the person detected by the LiDAR sensor210 (see FIG. 2B). Camera server 225 communicates the determinedpositions to the cluster servers 248.

Central server 240 may send a request to the cluster servers 248 to sendthe determined positions of the people during the first time window 332in order to determine whether those people have picked up any itemduring the first time window 332. This process is described in moredetail in corresponding descriptions of FIGS. 6A-6C.

Modifications, additions, or omissions may be made to method 342 a-bdepicted in FIGS. 3X and 3Y. Method 342 a-b may include more, fewer, orother steps. For example, steps may be performed in parallel or in anysuitable order. While discussed as particular components of camerasubsystem 242 performing the steps, any suitable component of camerasubsystem 242 may perform one or more steps of the method.

III. Light Detection and Ranging (LiDAR) Subsystem

Certain embodiments of tracking system 132 include a LiDAR subsystem204. FIGS. 4A-4D show the LiDAR subsystem 204 and its operation withintracking system 132. Generally, LiDAR subsystem 204 uses LiDAR sensorsand a LiDAR server to track the positions of people and/or objectswithin a physical space. LiDAR subsystem 204 may be used on its own orin conjunction with other subsystems (e.g., camera subsystem 202) totrack the positions of people and/or objects in the space.

FIG. 4A shows an example LiDAR subsystem 204. As seen in FIG. 4A, LiDARsubsystem 204 includes a LiDAR array 400 and a LiDAR server 230.Generally, LiDAR sensors 405 in LiDAR array 400 detect the presence ofpeople and/or objects within a space and determine coordinates for thesepeople and/or objects. LiDAR server 230 processes these coordinates todetermine the physical positions of the people and/or objects in thespace.

LiDAR array 400 is an array of LiDAR sensors 405. LiDAR array 400 may bepositioned above a physical space to detect the presence and positionsof people and/or objects within the space. In the example of FIG. 4A,LiDAR array 400 is a 3×4 array of LiDAR sensors 405. LiDAR array 400includes any suitable number of LiDAR sensors 405 arranged in an arrayof any suitable dimensions.

Each LiDAR sensor 405 detects the presence of people and/or objectswithin a portion of the physical space. Generally, LiDAR sensors 405emit light pulses into the space. These light pulses are reflected backtowards the LiDAR sensors 405 when the light pulses contact peopleand/or objects in the space. The LiDAR sensor 405 tracks characteristicsof the reflected light pulses, such as the return times of the lightpulses and the wavelength of the return light pulses, to detect thepresence of people and/or objects within the physical space. LiDARsensors 405 may also determine coordinates for the detected peopleand/or objects. LiDAR sensors 405 communicate the coordinates for thedetected people and/or objects to LiDAR server 230.

LiDAR sensors 405 may be communicatively coupled to LiDAR server 230 inany suitable manner. For example, LiDAR sensors 405 may be hardwired toLiDAR server 230. As another example, LiDAR sensors 405 may wirelesslycouple to LiDAR server 230 using any suitable wireless standard (e.g.,WiFi). LiDAR sensors 405 communicate coordinates for detected peopleand/or objects through the communication medium to LiDAR server 230.

FIG. 4B shows a LiDAR sensor 405 communicating coordinates 410 to LiDARserver 230. Generally, LiDAR sensor 405 analyzes characteristics ofreflected light pulses to determine the coordinates 410 of people and/orobjects within the space. LiDAR sensor 405 communicates thesecoordinates 410 to LiDAR server 230 for further processing. In theexample of FIG. 4B, LiDAR sensor 405 detects coordinates 410 for atleast two people and/or objects in the space. The coordinates 410 forthese people and/or objects are (x₁, y₁) and (x₂, y₂). LiDAR sensor 405communicates these coordinates 410 to LiDAR server 230 for furtherprocessing.

FIG. 4C illustrates the general operation of LiDAR server 230. As seenin FIG. 4C, LiDAR server 230 processes coordinates 410 received from theLiDAR sensors 405. LiDAR server 230 assigns coordinates 410 to timewindows 332 in a similar manner as camera server 225 assigns frame data330 to time windows 332. For example, LiDAR server 230 may assigncoordinates 410 to particular time windows 332 based on the time thatLiDAR server 230 received the coordinates 410 from LiDAR sensor 405.

LiDAR server 230 may process the coordinates 410 assigned to a timewindow 332 to determine the physical position of people and/or objectswithin the space. In the example of FIG. 4C, LiDAR server 230 receivescoordinates 410 for two people from two different LiDAR sensors 405. OneLiDAR sensor 405 provides coordinates 410 (x₁, y₁) and (x₂, y₂) for thetwo people, respectively. Another LiDAR sensor 405 provides coordinates410 (x₁, y₁) and (x₂, y₂) for the same two people, respectively. As withcamera client 220 and camera server 225, the subscripts on thesecoordinates 410 are not meant to indicate that these coordinates 410have the same value, but, rather, that these are the first and secondcoordinates 410 provided by a particular LiDAR sensor 405.

LiDAR server 230 uses these coordinates 410 to determine the physicalposition of people within the space. As with the camera server 225,LiDAR server 230 may determine that the coordinates 410 provided by twodifferent LiDAR sensors 405 correspond to the same person within thephysical space. In response, LiDAR server 230 may take these coordinates410 and use homography to determine a position of the person within thephysical space in a particular time window 332. In the example of FIG.4C, LiDAR server 230 uses coordinates 410 to determine the position of afirst person during the time window 332 to be (x₃, y₃). LiDAR server 230also uses coordinates 410 to determine the physical position of a secondperson during the time window 332 to be (x₄, y₄). LiDAR server 230communicates these physical positions to central server 240 for furtherprocessing.

FIG. 4D shows a method 415 for the operation of the LiDAR subsystem 204in the tracking system 132. Generally, LiDAR subsystem 204 performsmethod 415 to determine the positions of people and/or objects within aphysical space.

LiDAR sensor 405 determines coordinates 410 of detected people andcommunicates these coordinates 410 to LiDAR server 230. LiDAR sensor 405may determine these coordinates 410 by emitting a light pulse andanalyzing characteristics of the light pulse when that light pulse isreflected back to LiDAR sensor 405. For example, LiDAR sensor 405 mayanalyze the return time of the reflected light pulse and/or thewavelength of the reflected light pulse to determine whether a person ispresent in the physical space and the coordinates 410 of that person.

LiDAR server 230 analyzes the coordinates 410 from LiDAR sensor 405 todetermine the positions of people within the physical space during afirst time window 332 in step 416. LiDAR server 230 then communicatesthese positions to central server 240. LiDAR sensor 405 may subsequentlydetermine the coordinates 410 of detected people and communicate thesecoordinates 410 to LiDAR server 230. LiDAR server 230 may againdetermine the positions of these people in a subsequent time window 332and communicate these positions to central server 240 in step 418.

As with the camera subsystems 202, central server 240 may use thesepositions to determine which person removed an item 130 from the spaceduring the particular time window 332. The operation of central server240 will be described in more detail using FIG. 6A through FIG. 6C.

Modifications, additions, or omissions may be made to method 415depicted in FIG. 4D. Method 415 may include more, fewer, or other steps.For example, steps may be performed in parallel or in any suitableorder. While discussed as components of LiDAR subsystem 204 performingthe steps, any suitable component of tracking system 132, such ascentral server 240 for example, may perform one or more steps of themethod.

IV. Weight Subsystem

Tracking system 132 includes a weight subsystem 206 that includes weightsensors 215 and weight server 235. Generally, weight sensors 215 detectthe weights of items positioned above or near the weight sensors 215.The weight sensors 215 may be positioned on an unconventional rack 115that holds items. Weight server 235 tracks the weights detected byweight sensors 215 to determine if and when items 130 are removed fromthe rack 115. The weight sensors 215, rack 115, and weight server 235will be described in more detail using FIGS. 5A-5J.

FIG. 5A illustrates an example weight sensor 500 of weight subsystem206. As seen in FIG. 5A, weight sensor 500 includes plates 510A and510B, load cells 505A, 505B, 505C, and 505D, and wires 515A, 515B, 515C,515D, and 520. Generally, the components of weight sensor 500 areassembled so that weight sensor 500 can detect a weight of items 130positioned above or near weight sensor 500.

Plates 510 form surfaces that distribute the weight of items 130 acrossthe surfaces. Plates 510 may be made of any suitable material, such as,for example, metal and/or plastic. Items 130 may be positioned above ornear plates 510 and the weight of these items 130 may be distributedacross plates 510.

Load cells 505 are positioned between plates 510A and 510B. Load cells505 produce electrical signals based on the weight experienced by theload cells 505. For example, load cells 505 may be transducers thatconverts an input mechanical force (e.g., weight, tension, compression,pressure, or torque) into an output electrical signal (e.g., current orvoltage). As the input force increases, the output electrical signal mayincrease proportionally. Load cells 505 may be any suitable type of loadcell (e.g., hydraulic, pneumatic, and strain gauge). Although load cells1310 are illustrated as being cylindrical in shape, they may be anysuitable size and shape that is appropriate for the particularimplementation contemplated.

The signals from load cells 505 may be analyzed to determine an overallweight of items 130 positioned above or near weight sensor 500. Loadcells 505 may be positioned such that the weight of items 130 positionedabove or near weight sensor 500 is evenly distributed to each load cell505. In the example of FIG. 5A, load cells 505 are positionedsubstantially equidistant from corners of plates 510A and 510B. Forexample, load cell 505A is positioned a distance d1 from a corner ofplates 510A and 510B. Load cell 505B is positioned a distance d2 from acorner of plates 510A and 510B. Load cell 505C is positioned a distanced3 from a corner of plates 510A and 510B. Load cell 505D is positioned adistance d4 from a corner of plates 510A and 510B. Distances d1, d2, d3and d4 may be substantially equal to each other. This disclosurecontemplates distances differing by 5 to 10 millimeters and still beingconsidered substantially equal to each other. By positioning load cells505 substantially equal distances from corners of plates 510A and 510B,the weight of items positioned above or near weight sensor 500 is evenlydistributed across the load cells 505. As a result, the total weight ofitems positioned above or near weight sensor 500 can be determined bysumming the weights experienced by the individual load cells 505.

Load cells 505 communicate electric signals that indicate a weightexperienced by the load cells 505. For example, the load cells 505 mayproduce an electric current that varies depending on the weight or forceexperienced by the load cells 505. Each load cell 505 is coupled to awire 515 that carries the electric signal. In the example of FIG. 5A,load cell 505A is coupled to wire 515A; load cell 505B is coupled towire 515B; load cell 505C is coupled to wire 515C; and load cell 505D iscoupled to wire 515D. Wires 515 are grouped together to form wire 520that extends away from weight sensor 500. Wire 520 carries the electricsignals produced by load cells 505 to a circuit board that communicatesthe signals to weight server 235.

Weight sensor 500 may be disposed in an unconventional rack 115 designedto hold items. FIG. 5B shows an example rack 525. As seen in FIG. 5B,rack 525 includes a base 530, one or more panels 535, and one or moreshelves 540. Generally, base 530 is at the bottom of rack 525 and formsa foundation for the other components of rack 525. Panels 535 extendvertically upwards from base 530. Shelves 540 couples to panels 535and/or base 530. For example, two shelves 540 may couple to a panel 535and extend away from panel 535. Generally, panels 535 and base 530 allowshelves 540 to hold the weight of items positioned on shelves 540.Weight sensors 500 may be disposed within shelves 540 to detect theweight of items positioned on shelf 540.

FIG. 5C shows an exploded view of rack 525. As seen in FIG. 5C, base 530is formed using several surfaces 532. Surface 532A forms a bottomsurface of base 530. Surfaces 532B and 532D form the sides of base 530.Surface 532C forms a back surface of base 530. Surface 532E forms a topsurface of base 530. This disclosure contemplates base 530 being formedusing any suitable materials such as, for example, wood, metal, glass,and/or plastic. Surface 532A may be coupled to surfaces 532B, 532C, and532D. Surface 532B may be coupled to surfaces 532A, 532E, and 532C.Surface 532C may be coupled to surfaces 532A, 532B, 532D, and 532E.Surface 532D may be coupled to surfaces 532A, 532C, and 532E. Surface532E may be coupled to surfaces 532B, 532C, and 532D. Surfaces 532B,532C, and 532D extend upwards from surface 532A. Generally, surfaces532A, 532B, 532C, 532D, and 532E form a box structure around a space542. Base 530 includes a drawer 545 that can open to allow access intothat space 542. Drawer 545 is positioned within the space 542. Whendrawer 545 is closed, base 530 may form an enclosure around the space542. When drawer 545 is open, access to the space 542 may be providedthrough the open drawer 545. In certain embodiments, a door may be usedto provide access to space 542 rather than drawer 545.

Surface 532E defines a cavity 534 that also allows access into the space542. Generally, cavity 534 allows wires 520 from weight sensors 500 toextend into the space 542.

Panels 535 extend upwards from base 530. Panels 535 may be formed usingany suitable materials, such as for example, wood, metal, and/orplastic. As seen in FIG. 5C, panels 535 define one or more cavities 550that extend along the width of panels 535. Cavities 550 allow wires 520from weight sensors 500 to extend into a space 552 defined by panels535. Generally, space 552 is a hollow interior of panel 535. Wires 520extend through cavity 550 and down space 552 towards cavity 534. In thismanner, wires 520 may be run from weight sensors 500 down to space 542in base 530. Each cavity 550 may correspond to a shelf 540 that couplesto panel 535.

Each shelf 540 couples to panel 535 and/or base 530. Weight sensors 500are disposed in the shelf 540. A shelf 540 may couple to panel 535 suchthat the wires 520 of the weight sensors 500 disposed in the shelf 540can run from the weight sensors 500 through a cavity 550 into space 552.These wires 520 then run down space 552 and through cavity 534 intospace 542.

FIGS. 5D and 5E illustrate an example shelf 540. FIG. 5D shows a frontview of shelf 540. As seen in FIG. 5D, shelf 540 includes a bottomsurface 560A, a front surface 560B, and a back surface 560C. Bottomsurface 560A is coupled to front surface 560B and back surface 560C.Front surface 560B and back surface 560C extend upwards from bottomsurface 560A. Multiple weight sensors 500 are positioned on bottomsurface 560A between front surface 560B and back surface 560C. Eachweight sensor 500 is positioned to detect a weight of items 130positioned within certain regions 555 of shelf 540. Each region 555 maybe designated using dividers 558. Items placed within a particularregion 555 will be detected and weighed by the weight sensor 500 forthat region 555. This disclosure contemplates shelf 540 being made usingany suitable material such as, for example, wood, metal, glass, and/orplastic. Wires 515 and 520 have not been illustrated in FIG. 5D so thatthe structure of shelf 540 can be shown clearly, but their omission fromFIG. 5D should not be interpreted as their removal. This disclosurecontemplates that wires 515 and 520 are present and connected to weightsensors 500 in the example of FIG. 5D.

FIG. 5E shows a back view of shelf 540. As seen in FIG. 5E, back surface560C defines a cavity 562. Wires 520 of weight sensors 500 extend fromthe weight sensors 500 through cavity 562. Generally, back surface 560Cof shelf 540 is coupled to panel 535 such that cavity 562 is at leastpartially aligned with cavity 550 in the panel 535. In this manner,wires 520 can run from weight sensors 500 through cavity 562 and throughcavity 550.

In certain embodiments, weight sensor 500 is positioned in shelf 540such that weight sensor 500 detects the weight of items positionedwithin a particular region 555 of shelf 540. As seen in the examples ofFIGS. 5D and 5E, shelf 540 includes four regions 555 that are positionedabove four weight sensors 500. Each weight sensor 500 detects the weightof items positioned within their corresponding regions 555. Due to thepositioning of weight sensors 500, a weight sensor 500 may not beaffected by the weight of items 130 positioned in regions 555 that donot correspond to that weight sensor 500.

FIG. 5F shows an example base 530. As seen in FIG. 5F, base 530 may alsoaccommodate weight sensors 500. For example, weight sensors 500 may bepositioned on a top surface 532E of base 530. Wires 520 for these weightsensors 500 may run from the weight sensors 500 through cavity 534 intospace 542. As a result, items may be positioned on base 530 and theirweights may be detected by weight sensors 500.

A circuit board 565 is positioned in space 542. Circuit board 565includes ports to which wires 520 from the weight sensors 500 of rack525 connect. In other words, circuit board 565 connects to wires 520from weight sensors 500 positioned on base 530 and on shelves 540. Thesewires 520 enter space 542 through cavity 534 and connect to circuitboard 565. Circuit board 565 receives the electric signals produced bythe load cells 505 of the weight sensors 500. Circuit board 565 thencommunicates signals to weight server 235 indicating the weightsdetected by the weight sensors 500. Drawer 545 may open to allow accessto space 542 and to circuit board 565. For example, drawer 545 may beopened so that circuit board 565 may be serviced and/or repaired.

FIG. 5G shows an example circuit board 565. As seen in FIG. 5G, circuitboard 565 includes a processor 566 and multiple ports 568. Generally,ports 568 couple to wires 520 from weight sensors 500. This disclosurecontemplates circuit board 565 including any suitable number of ports568 to connect to the wires 520 from the weight sensors 500 of rack 525.Processor 566 receives and processes the signals from ports 568.

Circuit board 565 may communicate signals to weight server 235 throughany suitable medium. For example, circuit board 565 may communicatesignals to weight server 230 through an ethernet connection, a wirelessconnection (e.g., WiFi), a universal serial bus connection, and/or aBluetooth connection. Circuit board 565 can automatically select aconnection through which to communicate signals to weight server 235.Circuit board 565 may choose the connection based on priority. Forexample, if the ethernet connection is active, circuit board 565 mayselect the ethernet connection for communicating with weight server 235.If the ethernet connection is down and the wireless connection isactive, circuit board 565 may choose the wireless connection tocommunicate with weight server 235. If the ethernet connection and thewireless connection are down and the universal serial bus connection isactive, circuit board 565 may select the universal serial bus connectionto communicate with weight server 235. If the ethernet connection, thewireless connection, and the universal serial bus connection are downand the Bluetooth connection is active, circuit board 565 may select theBluetooth connection to communicate with weight server 235. In thismanner, circuit board 565 has improved resiliency because circuit board565 may continue to communicate with weight server 235 even if certaincommunication connections go down.

Circuit board 565 may receive electrical power through variousconnections. For example, circuit board 565 may include a power port 570that supplies electrical power to circuit board 565. An electrical cablethat plugs into an electrical outlet may couple to power port 570 tosupply electrical power to circuit board 565. Circuit board 565 may alsoreceive electrical power through the ethernet connection and/or theuniversal serial bus connection.

FIG. 5H shows a signal 572 produced by the weight sensor 500. As seen inFIG. 5H the signal 572 begins by indicating a certain weight detected bythe weight sensors 500. Around time t₁ an item positioned above theweight sensor 500 is taken. As a result, the weight sensor 500 detects adrop in the weight and the signal 572 experiences a corresponding drop.Beyond time t₁, the signal 572 continues to hover around the lowerweight because the item 130 was removed. This disclosure contemplatesthat the signal 572 may include noise introduced by the environment suchthat the signal 572 is not a perfectly straight or smooth signal.

FIG. 5I shows an example operation of weight server 235. As seen in FIG.5I, weight server 235 receives a signal 572 from a weight sensor 500 attime to indicating a weight w₀. Similar to camera server 225, weightserver 235 may assign this information to a particular time window 332Abased on the indicated time of t₀. Later, weight server 235 may receivea signal 572 from the weight sensor 500 indicating that at time t₁, anew weight w₁ is detected. Weight w₁ may be less than weight w₀, therebyindicating that an item 130 may have been removed. Weight server 235assigns the information to a subsequent time window 332C based on thetime indicated at t₁.

Weight server 235 may implement an internal clock 304E that issynchronized with the internal clocks 304 of other components oftracking system 132 (e.g., camera clients 220, camera server 225, andcentral server 240). Weight server 235 may synchronize the internalclock 304E using a clock synchronization protocol (e.g., Network TimeProtocol and/or Precision Time Protocol). Weight server 235 may useclock 304E to determine the times at which signals 572 from weightsensors 500 were received and assign these signals 572 to theirappropriate time windows 332.

In certain embodiments, time windows 332 in weight server 235 arealigned with time windows 332 in camera clients 220, camera server 225,and/or central server 240. For example, time window 332A in weightserver 235 may have the same start time (T0) and end time (T1) as timewindow 332A in camera server 225 in the example of FIG. 3J. In thismanner, information from different subsystems of tracking system 132 maybe grouped according to the same time windows 332, which allows thisinformation to be correlated to each other in time.

Similar to camera server 225, weight server 235 may process theinformation in the time windows 332 sequentially when the time windows332 are ready for processing. Weight server 235 may process theinformation in each time window 332 to determine whether an item 130 wasremoved during that particular time window 332. In the example of FIGURESI, when weight server 235 processes the third time window 332C, weightserver 235 may determine that sensor 1 500 detected that two items weretaken during time window 3 332C; thereby, resulting in the weight dropfrom w₀ to w₁. Weight server 235 may make this determination bydetermining a difference between w₀ and w₁. Weight server 235 may alsoknow (e.g., through a lookup table) the weight of an item 130 positionedabove or near weight sensor 500. Weight server 235 may divide thedifference between w₀ and w₁ to determine the number of items 130removed. Weight server 235 may communicate this information to centralserver 240 for further processing. Central server 240 may use thisinformation along with the tracked positions of people within the spaceto determine which person in the space removed the items 130.

FIG. 5J shows an example method 580 for operating weight subsystem 206.Generally, various components of weight subsystem 206 perform method 580to determine when certain items 130 were taken.

Weight sensor 215 detects the weight experienced 582 above or aroundweight sensor 215 and communicates the detected weight 582 through anelectric signal 572 to weight server 235. Weight server 235 may analyzethe signals 572 from weight sensor 215 to determine a number 584 ofitems 130 that were taken during a first time window 332. Weight server235 may communicate the determination to central server 240. Weightsensor 215 may subsequently detect a weight 586 experienced by weightsensor 215 and communicate that weight 586 to weight server 235. Weightserver 235 may analyze that weight 586 to determine a number 588 ofitems 130 that were taken during a second time window 332. Weight server235 may communicate that determination to central server 240. Centralserver 240 may track whether items 130 were taken during particular timewindows 332. And if so, central server 240 may determine which person inthe space took those items 130.

Modifications, additions, or omissions may be made to method 580depicted in FIG. 5J. Method 580 may include more, fewer, or other steps.For example, steps may be performed in parallel or in any suitableorder. While discussed as various components of weight subsystem 206performing the steps, any suitable component of tracking system 132,such as central server 240 for example, may perform one or more steps ofthe method.

FIG. 5K illustrates a rear view of a shelf 540, which may be similar tothe shelf 540 illustrated in FIG. 5E. As seen in FIG. 5K, the shelf 540includes rear surface 560 c that defines cavity 562. Wires 520 a-d ofweight sensors 500 a-d, respectively, extend from the weight sensors 500a-d through cavity 562. Weight sensors 500 a-d are instances of weightsensor 500 of FIG. 5A.

Each weight sensor 500 a-d is positioned in the shelf 540 such that eachweight sensor 500 a-d detects the weight of items positioned within aparticular region 555 a-d of shelf 540, respectively. For example,weight sensor 500 a detects the weight of item(s) positioned withinregion 555 a, weight sensor 500 b detects the weight of item(s)positioned within region 555 b, weight sensor 500 c detects the weightof item(s) positioned within region 555 c, and weight sensor 500 ddetects the weight of item(s) positioned within region 555 d. Althoughshelf 540 is shown with four regions 555 a-d, this disclosurecontemplates any suitable number and combination of regions 555 andweight sensors 500 positioned in a shelf 540.

Each weight sensor 500 is assigned a particular address number 804indicating the position of each weight sensor 500 within the store 100(see FIG. 1B). For example, an address number 804 of a weight sensor 500may specify a number associated with a shelf 540 and a number associatedwith a zone/region 555. For instance, an address number 804 of a weightsensor 500 positioned in a first region 555 in a first shelf 540 mayspecify that the weight sensor 500 is positioned in shelf 1-region 1. Inanother instance, an address number 804 of a weight sensor 500positioned in a second region 555 in a third shelf 540 may specify thatthe weight sensor 500 is positioned in shelf 3-region 2. In theillustrated example shelf 540 in FIG. 5K, weight sensor 500 a isassigned an address number 804 a which indicates that the weight sensor500 a is positioned in region 555 a, weight sensor 500 b is assigned anaddress number 804 b which indicates that the weight sensor 500 b ispositioned in region 555 b, weight sensor 500 c is assigned an addressnumber 804 c which indicates that the weight sensor 500 c is positionedin region 555 c, and weight sensor 500 d is assigned an address number804 d which indicates that the weight sensor 500 d is positioned inregion 555 d.

Each weight sensor 500 is configured to transmit a signal that includesan address number 804 of that weight sensor 500 and sensor dataindicating a weight of an item to its corresponding weight board 800through wires 520. In the illustrated example shelf 540 in FIG. 5K, aweight sensor 500 a is connected to a weight board 800 a through wires520 a, weight sensor 500 b is connected to a weight board 800 b throughwires 520 b, weight sensor 500 c is connected to a weight board 800 cthrough wires 520 c, and weight sensor 500 d is connected to a weightboard 800 d through wires 520 d.

Weight board 800 is generally a board that is configured to processsignals generated from a weight sensor 500. Operations of the weightboard 800 are described in detail in the corresponding description ofFIG. 5L. In brief, the weight board 800 is configured to digitize asignal received from a weight sensor 500. The weight board 800 is alsoconfigured to assign an address number 804 to its corresponding weightsensor 500. In the illustrated example shelf 540 in FIG. 5K, weightboard 800 a is used to assign address number 804 a to weight sensor 500a, weight board 800 b is used to assign address number 804 b to weightsensor 500 b, weight board 800 c is used to assign address number 804 cto weight sensor 500 c, and weight board 800 d is used to assign addressnumber 804 d to weight sensor 500 d.

As further seen in FIG. 5K, wires 806 a-d extending from weight boards800 a-d are aggregated into a bus wire 802. Bus wire 802 connects to aport on the circuit board 565 with reference to FIG. 5M. Weight boards800 a-d communicate the processed signals (received from weight sensors500 a-d) through the bus wire 802 to the circuit board 565. Wires 806a-d may be aggregated into the bus wire 802 in any suitable manner,e.g., bundling wires together. Wires 806 a-d may be aggregated into thebus wire 802 at any point before connecting to a port 568 of the circuitboard 565.

FIG. 5L illustrates an example weight board 800. In one embodiment, theweight board 800 may include a processor 808, a temperature sensor 810,a Liquid Crystal Display (LCD) port 812, a keypad port 814, a connectorport 816, a power port 818, universal serial bus communication port 820,and port 822. In other embodiments, the weight board 800 may not haveall of the components listed and/or may have other elements instead of,or in addition to, those listed above.

As seen in FIG. 5L, wire 520 extending from a weight sensor 500 isconnected to the port 822 that is operably coupled with processor 808.Thus, a signal 572 and an address number 804 from the weight sensor 500are transmitted to the processor 808 via the wire 520 and port 822. Theprocessor 808 may be one of the processing units of the processor 705described in FIG. 7.

Processor 808 may include any electronic circuitry, including, but notlimited to microprocessors, application specific integrated circuits(ASIC), application specific instruction set processor (ASIP), and/orstate machines. Processor 808 may be 8-bit, 16-bit, 32-bit, 64-bit or ofany other suitable architecture. Processor 808 may include an arithmeticlogic unit (ALU) for performing arithmetic and logic operations,processor registers that supply operands to the ALU and store theresults of ALU operations, and a control unit that fetches instructionsfrom memory and executes them by directing the coordinated operations ofthe ALU, registers and other components. Processor 808 may include otherhardware that operates software to control and process information.Processor 808 executes software stored on memory to perform any of thefunctions described herein. Processor 808 controls the operation andadministration of weight board 800 by processing information receivedfrom components of the weight board 800, such as the temperature sensor810, LCD port 812, keypad port 814, connector port 816 a power port 818,universal serial bus communication port 820, and port 822. Processor 808may be a programmable logic device, a microcontroller, a microprocessor,any suitable processing device, or any suitable combination of thepreceding. Processor 808 is not limited to a single processing deviceand may encompass multiple processing devices.

The processor 808 is configured to digitize the signal 572 received fromthe weight sensor 500. In one embodiment, the processor 808 may includeor use any Analog-to-Digital Converter (ADC) module or circuitry toconvert an analog signal 572 (e.g., voltage or current) representing aweight of an item to a digital signal 574. The digital signal 574represents digitized sensor data of the weight sensor 500. The processor808 may also timestamp when it receives the signal 574. The weight board800 then transmits the signal 574 and the address number 804 to acircuit board 565 (see FIG. 5M) through the wire 806 extending from theconnector port 816.

Temperature sensor 810 may be any temperature sensor module capable ofcapturing the temperature of its surroundings. The temperature sensor810 is communicatively coupled with the processor 808. The temperaturesensor 810 is used to register the temperature of the environment aroundthe weight sensor 500 and report that information to the processor 808.The processor 808 uses the information received from the temperaturesensor 810 to determine correlations between fluctuations in a weightsensor sensitivity of the weight sensor 500 and temperature changes. Theweight sensor sensitivity of a weight sensor 500 may be determined by aratio of an electrical signal change (e.g., a current or voltage change)over a weight change detected by that weight sensor 500.

In some cases, a temperature change can affect a weight sensorsensitivity of a weight sensor 500, such that it may convert anelectrical signal change to a weight change that is not expected fromthe weight sensor 500 using a default sensor sensitivity of the weightsensor 500. The default sensor sensitivity of the weight sensor 500 is aratio of an electrical signal change over a weight change that isexpected from the weight sensor 500. For example, assume that undernormal environmental conditions, e.g., temperature 20-22° C., the weightsensor 500 converts a 4 Volt (V) change to a 2 pounds (lb.) weightchange. In other words, under normal environmental conditions, thedefault sensor sensitivity of the weight sensor 500 is 2 (V/lb.).However, due to environmental changes, e.g., temperature changes, theweight sensor 500 may convert the 4 V voltage change to a 4 lb. weightchange.

In this exemplary case, the weight sensor sensitivity is 1 (V/lb.) whichis shifted from the default sensor sensitivity of the weight sensor 500.Therefore, the temperature sensor 810 is used to keep track oftemperature changes in order to detect such scenarios where a weightsensor sensitivity shifts from a default sensor sensitivity beyond aconfigurable threshold range. As such, the information received from thetemperature sensor 810 is further used to provide a training dataset foran item counting algorithm to accurately count how many items areremoved from or put on a weight sensor 500 at different temperatures.The item counting algorithm is described in detail in correspondingdescriptions of FIGS. 5N and 5O.

As further seen in FIG. 5L, the weight board 800 also includes the LCDport 812. As described above with respect to FIG. 5K, each weight sensor500 is assigned a particular address number 804. In one embodiment, anaddress number 804 of a weight sensor 500 can be configured and assignedto that weight sensor 500 from an LCD (not shown) that is connected tothe LCD port 812. The LCD may serve as a user interface to show weightsensor information, such as analog signal 772, digital signal 574,address number 804, etc.

In one embodiment, the address number 804 of the weight sensor 500 maybe inputted from a keypad (not shown) that is connected to the keypadport 814. For example, when assigning an address number 804 to a weightsensor 500, an operator can use the keypad to set the address number 804to the weight sensor 500. In another example, the operator can assign anaddress number 804 to a weight sensor 500 from a touch screen LCD.

Weight board 800 may receive electrical power through variousconnections. For example, the weight board 800 may include the powerport 818 that supplies electrical power to the weight board 800. Anelectrical cable that plugs into an electrical outlet may couple to thepower port 818 to supply electrical power to the weight board 800.Weight board 800 may also receive electrical power using other methods,such as through the ethernet connection, the universal serial busconnection 820, etc.

In one embodiment, weight board 800 may include other components such asan amplifier module to amplify incoming signals 572 from the weightsensor 500, an oscillating component to set the speed of processor 808,a memory card port to store weight sensor information into a memorycard, among other components.

By processing the incoming signals 572 in the weight board 800, theprocessing of the incoming signals 572 is performed closer to the weightsensor 500. As such, the incoming signals 572 are traveling a shorterdistance, thus, they are less noisy. In other words, by processing theincoming signals 572 from the weight sensor 500 in the weight board 800,an edge computing is leveraged to produce more accurate results (i.e.,more accurate weights of items on the weight sensors 500). Thus, theimproved system described herein provides additional technicaladvantages and facilitates the practical application of providingaccurate weight measurements.

FIG. 5M illustrates an example circuit board 565. The illustratedexample circuit board 565 in FIG. 5M may be similar to the examplecircuit board 565 of FIG. 5G. The example circuit board 565 in FIG. 5Mcorresponds to the circuit board 254 illustrated in FIG. 2B.

As described above with respect to FIG. 5K, the bus wire 802 includesthe aggregated wires 806 a-d extending from weight boards 800 a-d to aport 568 on the circuit board 565. The bus wire 802 carries one or moresignals 576 that include digital signal(s) 574 and address number(s) 804of one or more weight sensors 500. From the one or more signals 576, thecircuit board 565 determines from which weight board 800 (or whichweight sensor 500) a signal 576 is sent based on the address numbers 804of weight sensors 500. For example, with reference to FIG. 5K, if thecircuit board 565 receives a first signal 576 that includes a firstaddress number 804 a, the circuit board 565 determines that the firstsignal 576 is sent from the first weight board 800 (or originated fromthe first weight sensor 500 a).

In one embodiment, the circuit board 565 is configured to perform anerror detection step on the signals 576. In performing the errordetection step, when the circuit board 565 receives one or more signals576, it determines whether each incoming signal 576 is valid. Forexample, the circuit board 565 determines that an incoming signal 576 isvalid if the incoming signal 576 includes an address number 804 and asignal 574 that indicates weight information of item(s). If the incomingsignal 576 does not include any of these portions, the circuit board 565determines that the incoming signal 576 is not valid. If, however, theincoming signal 576 includes both of these portions, the circuit board565 determines that the incoming signal 576 is valid.

Circuit board 565 is assigned an Internet Protocol (IP) address 824which is used to identify the circuit board 565. For example, thecircuit board 565 may be assigned the IP address 824 at the weightserver 235 (see FIG. 2B). Throughout the store, each circuit board 565is assigned a different IP address 824. As such, based on the IPaddresses 824 of the circuit boards 565, the location and identity ofeach circuit board 565 may be determined.

Referring back to FIG. 2B, the circuit board 565 may transmit signals576 to a server among the cluster servers 248 (e.g., via a UDP/TCPbridge module) through a wireless communication, such as WiFi.

Wirelessly communicating the signals 576 provides additional technicaladvantages and facilitates practical applications of reducing wiringcomplexity of the circuit boards 565 to the cluster servers 248.Furthermore, since each weight sensor 500 is assigned a particularaddress number 804 and signals 576 are transmitted wirelessly, rackswithin the store can be re-arranged more flexibly according to thedesired layout of the store because the circuit boards 564 are notcommunicating data to the cluster servers 248 through wires. Once theracks are positioned according to the desired layout, the addressnumbers 804 of the weight sensors 500 are used to determine the newpositions of the weight sensors 500.

The cluster servers 248 store and maintain the signals 576, and forwarda particular set of signals 576 to the weight server 235 upon receivinga request from the weight server 235 to send the particular set ofsignals 576 from a particular time window (e.g., a timestamps range), aparticular circuit board 565 (e.g., identified by an IP address 824), aparticular address number 804, etc. The weight server 235 determinesfrom which weight sensor 500 and at which timestamp a particular signal576 is generated based on the address numbers 804 of the weight sensors500 and a timestamp label of the particular signal 576. As such, thetracking system 132 b can improve an item position detection andtracking system 132.

FIG. 5N illustrates an example flow chart of a method 900 for anadaptive item counting algorithm. Generally, various components ofweight subsystems 206 or 250 (see FIGS. 2A and 2B) perform method 900 todetermine an item quantity that is removed from or put on a weightsensor 500. In one embodiment, method 900 is implemented by processor705 executing software instructions or code (see FIG. 7). Aspects ofmethod 900 are explained above with respect to FIGS. 5A-5M, andadditional aspects are given below. In one embodiment, method 900 may beimplemented by a machine learning algorithm, such as Support VectorMachine (SVM), etc. In one embodiment, method 900 may be implemented bya plurality of neural network layers, convolutional neural networklayers, and the like.

Method 900 beings at step 902 when the weight server 235 receives afirst signal 574 indicative of a weight associated with at least one ofan item 130 (see FIG. 1B). In step 904, the weight server 235 detects afirst event corresponding to a weight change on the weight sensor 500,when a quantity of the item 130 is removed from (or, alternatively, puton) the weight sensor 500 (see FIG. 5A).

In step 906, the weight server 235 determines a weight decrease amountcorresponding to the weight decrease on the weight sensor 500. Forexample, the weight server 235 may determine the weight decrease amountby calculating a difference between an initial weight at the beginningof the first event and an end weight at the end of the first event. In acase where at least one of the item 130 is put on the weight sensor 500,the weight server 235 determines a weight increase amount correspondingto a weight increase on the weight sensor 500.

In step 908, the weight server 235 calculates a result from dividing theweight decrease amount (or, alternatively, weight increase amount) overa unit weight of the item 130. For example, the result may be calculatedby equation (1) below:

$\begin{matrix}{{result} = \frac{{Weight}_{end} - {Weight}_{Initial}}{{Weight}_{unit}}} & {{Equation}\mspace{14mu} (1)}\end{matrix}$

In step 910, the weight server 235 determines an item quantity value byidentifying the closest integer to the result. In a first example,assume that two of the item 130 are removed from the weight sensor 500;the unit weight of the item 130 is 2 pounds; and the result iscalculated as 4.2. As such, the result using the equation (1) will be2.1. In this particular example, the closest integer to the result is 2.

In a second example, assume that two of the item 130 are removed fromthe weight sensor 500; the unit weight of the item 130 is 2 pounds; andthe result is calculated as 4.6. As such, the result using the equation(1) will be 2.3. In this particular example, the closest integer to theresult is 2.

In step 912, the weight server 235 determines whether the result iswithin a first threshold range from the item quantity value. The firstthreshold range may be, for example, 20% or any other percentage range.As such, the first threshold range from the result in the first andsecond examples given in step 910 may be calculated from ±20% from theinteger value 2, i.e., 1.8 to 2.2.

If it is determined that the result is within the first threshold rangefrom the item quantity value, the method 900 proceeds to step 914 wherethe weight server 235 determines that a quantity of the item 130 withthe amount of the item quantity value is removed from (or,alternatively, put on) the weight sensor 500. If, however, it isdetermined that the result is not within the first threshold range fromthe item quantity value, the method 900 proceeds to step 916.

Continuing the first example described in step 910, since the result is2.1, the weight server 235 determines that the result is within thefirst threshold range, i.e., 1.8 to 2.2. Thus, the weight server 235determines that two of the item 130 are removed from the weight sensor500.

Continuing the second example described in step 910, since the result is2.3, the weight sensor 235 determines that the result is not within thefirst threshold range, i.e., 1.8 to 2.2, and thus the method proceeds toperform steps 916-920. The steps 916-920 may be performed by a patternrecognition technique to determine the item quantity that was removedfrom (or, alternatively, put on) the weight sensor 500.

In step 916, the weight server 235 compares a weight change pattern inthe first signal 572 with weight change patterns from a training datasetthat includes a set of historically observed signals, where each signalof the training dataset is associated (or labeled) with a quantity ofthe item 130 being removed from (or, alternatively, put on) the weightsensor 500. In this process, the weight server 235 may use any patternrecognition technique, such a pattern recognition from images (e.g.,plots of analog/digital signals), text (e.g., tables of analog/digitalsignal values), videos (e.g., streams of data feed of analog/digitalsignals), etc.

The training dataset may include any number of historically observedsignals. For example, a first historically observed signal from thetraining dataset may show a weight decrease when two of the item 130 areremoved from the weight sensor 500, and the first historically observedsignal is labeled with two of the item being removed; a secondhistorically observed signal from the training dataset may show a weightdecrease when three of the item 130 are removed from the weight sensor500, and the second historically observed signal is labeled with threeof the item being removed; and a third historically observed signal fromthe training dataset may show a weight decrease when one of the item 130is removed from the weight sensor 500, and the third historicallyobserved signal is labeled with one of the item being removed. Thetraining dataset may also include historically observed signals labeledwhen one or more items are put on a weight sensor 500.

In one embodiment, the historically observed signals from the trainingdataset may also be labeled with a temperature of the environment aroundthe weight sensor 500 when those signals were recorded. As such, theweight change pattern in the first signal 572 is compared with a set ofhistorically observed signals from the training dataset that arerecorded within a temperature range (e.g., within 2° C.) from thecurrent temperature surrounding the weight sensor 500 detected by thetemperature sensor 810 (see FIG. 5L).

In step 918, the weight server 235 finds a particular signal from thetraining dataset whose weight change pattern matches the weight changepattern of the first signal 572. In step 920, the weight server 235determines that a quantity of the item 130 associated with theparticular signal (found in the training dataset) is removed from (or,alternatively, put on) the weight sensor 500.

Modifications, additions, or omissions may be made to method 900depicted in FIG. 5N. Method 900 may include more, fewer, or other steps.For example, steps may be performed in parallel or in any suitableorder. While discussed as particular components of weight subsystems 206or 250 performing the steps, any suitable component of weight subsystems206 or 250 may perform one or more steps of the method.

FIG. 5O illustrates an example flow chart of a method 1000 forvalidating a result from an item counting process, such as the method900 of FIG. 5M. In one embodiment, method 1000 is implemented byprocessor 705 executing software instructions or code (see FIG. 7). Inone embodiment, method 1000 may be implemented by any machine learningalgorithm, such as SVM, etc. In one embodiment, method 1000 may beimplemented by a plurality of neural network layers, convolutionalneural network layers, and the like. In one embodiment, the method 1000may be performed separately from method 900 of FIG. 5M. In anotherembodiment, the methods 900 and 1000 may be performed together. Similarto method 900 in FIG. 5M, method 1000 can be adapted for both cases whena quantity of an item 130 is removed from or put on a weight sensor 500.

Method 1000 begins at step 1002 when the weight server 235 analyzes aresult from a first event corresponding to a quantity of an item 130being removed from (or, alternatively, put on) a weight sensor 500. Forexample, method 1000 analyzes the result from either step 914 or step920 of method 900 of FIG. 5M.

In step 1004, the weight server 235 calculates an instantaneous sensorsensitivity of the weight sensor 500. The instantaneous sensorsensitivity corresponds to a ratio of an electrical, i.e., voltage orcurrent change over a weight change during the first event. For example,if the weight sensor 500 detects a weight change with an amount of 2pounds (lb.), and converts the 2 pounds (lb.) to a voltage change withan amount of 4 Volt (V), the instantaneous sensor sensitivity of theweight sensor 500 is 2 V/lb.

In step 1006, the weight server 235 determines whether the instantaneoussensor sensitivity is within a second threshold range from a defaultsensor sensitivity of the weight sensor 500. The default sensorsensitivity of the weight sensor is a predetermined ratio of a voltagechange over a weight change that is expected from the weight sensor 500.The second threshold range may be, for example, 20% or any otherpercentage range.

If it is determined that the instantaneous sensor sensitivity is withinthe second threshold range from the default sensor sensitivity, themethod 1000 proceeds to perform steps 1008 and 1010. If, however, it isdetermined that the instantaneous sensor sensitivity is not within thesecond threshold range from the default sensor sensitivity, the method1000 proceeds to perform steps 1012 and 1014. In other words, in step1006, the weight server 235 determines whether the environmentalchanges, e.g., the temperature changes around the weight sensor 500 haveaffected the sensor sensitivity of the weight sensor 500 such that itdetects an inaccurate voltage change, i.e., determines whether theinstantaneous sensor sensitivity is shifted from the default sensorsensitivity beyond the second threshold range. For example, under normalenvironmental conditions, e.g. temperature between 20-22° C., it isexpected that the weight sensor 500 to have the default sensorsensitivity, e.g., 2.5 V/lb.

In a first example, assume that the instantaneous sensor sensitivity iscalculated to be 2 V/lb., the default sensor sensitivity is 2.1 V/lb.,and the second threshold range is 20%. In this particular example, theweight server 235 determines that the instantaneous sensor sensitivityis within the second threshold range from the default sensorsensitivity. As such, the method 1000 proceeds to step 1008.

In a second example, assume that the instantaneous sensor sensitivity iscalculated to be 2.5 V/lb., and the default sensor sensitivity is 2.1V/lb., and the second threshold range is 20%. In this particularexample, the weight server 235 determines that the instantaneous sensorsensitivity is not within the second threshold range from the defaultsensor sensitivity. As such, the method 1000 proceeds to step 1012.

In step 1008, the weight server 235 confirms that the quantity of theitem 130 indicated in the result (e.g., determined by the method 900) isremoved from (or, alternatively, put on) the weight sensor 500. In otherwords, the weight server 235 determines that at the time of the firstevent, environmental changes, e.g., temperature changes around theweight sensor 500 have not affected the sensor sensitivity of the weightsensor 500 such that the weight server 235 has detected an inaccurateitem quantity.

In step 1010, the weight server 235 records that the instantaneoussensor sensitivity to be used for a next event corresponding to a weightchange on the weight sensor 500. This is due to the determination thatthe instantaneous sensor sensitivity of the weight sensor 500 is notshifted from its default sensor sensitivity beyond the second thresholdrange, i.e., weight sensor 500 is not affected by the environmentalchanges.

In step 1012, the weight server 235 determines that a quantity of theitem 130 that is closest to an item quantity value using the defaultsensor sensitivity is removed from (or, alternatively, put on) theweight sensor 500. In other words, the weight server 235 determines thatat the time of first event, the environmental changes, e.g., temperaturechanges around the weight sensor 500 have affected the sensorsensitivity of the weight sensor 500 such that the weight server 235 hasdetected an inaccurate item quantity. As such, the weight server 235updates the result by re-calculating the result using the default sensorsensitivity of the weight sensor 500.

For example, assume that the instantaneous sensor sensitivity iscalculated to be 2.5 V/lb., and the default sensor sensitivity is 2.1V/lb., the second threshold range is 20%, and the result calculatedusing the default sensor sensitivity is 2.3 V/lb. In this particularexample, the closest integer to the result is 2. Thus, the weight server235 determines that two of the item 130 are removed from (or,alternatively, put on) the weight sensor 500 during the first event.

In step 1014, the weight server 235 records that the default sensorsensitivity to be used for a next event corresponding to a weight changeon the weight sensor 500. This is due to the determination that theinstantaneous sensor sensitivity of the weight sensor 500 is shiftedfrom its default sensor sensitivity beyond the second percentage range,i.e., weight sensor 500 is affected by the environmental changes.

In step 1016, the weight server 235 updates the instantaneous sensorsensitivity of the weight sensor 500 to be used for the next event. Theweight server 235 updates the instantaneous sensor sensitivity based onthe sensor sensitivity of the weight sensor 500 calculated during thefirst event. With this method, the weight server 235 keeps track ofchanges or shifts in the sensor sensitivity of the weight sensor 500. Assuch, in the next event, in step 1006, the weight server 235 comparesthe updated instantaneous sensor sensitivity with the default sensorsensitivity.

Modifications, additions, or omissions may be made to method 1000depicted in FIG. 5O. Method 1000 may include more, fewer, or othersteps. For example, steps may be performed in parallel or in anysuitable order. While discussed as particular components of weightsubsystems 206 or 250 performing the steps, any suitable component ofweight subsystems 206 or 250 may perform one or more steps of themethod.

V. Central Server

FIGS. 6A-6C show the operation of central server 240. Generally, centralserver 240 analyzes the information from the various subsystems (e.g.,camera subsystem 202, LiDAR subsystem 204, weight subsystem 206, etc.)and determines which person in a space removed which items from thespace. As discussed previously, these subsystems group information intotime windows 332 that are aligned across the subsystems. By groupinginformation into aligned time windows 332, central server 240 can findrelationships between information from disparate subsystems and gleanadditional information (e.g., which person removed which item 130). Insome embodiments, central server 240 also charges people for items theyremoved from the space when those people exit store 100.

FIGS. 6A and 6B show an example operation of central server 240. As seenin FIG. 6A, central server 240 receives information from various serversduring particular time windows. In the example of FIG. 6A, centralserver 240 receives the physical position of two people in the spacefrom camera server 225 during a first time window 332A. This disclosureuses capital ‘X’ and capital ‘Y’ to denote the physical coordinates 602of a person or object in the space and to distinguish the physicalcoordinates 602 of the person or object in the space determined bycamera server 225 and LiDAR server 230 from the local coordinatesdetermined by other components (e.g., coordinates 322 determined bycamera clients 220 and coordinates 410 determined by LiDAR sensors 405).

According to the camera server 225, the first person is at a physicalcoordinate 602 (X₁, Y₁), and the second person is at a physicalcoordinate 602 (X₂, Y₂). Additionally, central server 240 receives fromLiDAR server 230 the physical location of the two people. According tothe LiDAR server 230, the first person is at coordinate 602 (X₇, Y₇) andthe second person is at coordinate 602 (X₈, Y₈). Furthermore, centralserver 240 also receives information from weight server 235 during thefirst time window 332A. According to weight server 235, no items 130were taken during the first time window 332A.

This disclosure contemplates central server 240 using any suitableprocess for analyzing the physical position of people from camera server225 and LiDAR server 230. Although the coordinates 602 provided bycamera server 225 and LiDAR server 230 may differ from each other,central server 240 may use any appropriate process for reconciling thesedifferences. For example, central server 240 may use the coordinates 602provided by camera server 225 if the coordinates 602 provided by LiDARserver 230 do not differ from the coordinates 602 provided by cameraserver 225 by an amount that exceeds a threshold. In this manner, thecoordinates 602 provided by LiDAR sever 230 act as a check on thecoordinates 602 provided by camera server 225.

During a second time window 332B, central server 240 receives fromcamera server 225 the physical coordinates 602 of the two people.According to camera server 225, during the second time window 332B, thefirst person was at coordinate 602 (X₃, Y₃) and the second person was atcoordinate 602 (X₄, Y₄). During the second time window 332B, cameraserver 240 also receives the physical coordinates 602 of the two peoplefrom LiDAR server 230. According to the LiDAR server 230, the firstperson is at coordinate 602 (X₉, Y₉) and the second person is atcoordinate 602 (X₁₀, Y₁₀) during the second time window 332B.Additionally, central server 240 learns from weight server 235 that noitems 130 were taken during the second time window 332B.

During a third time window 332C, camera server 240 receives the physicalcoordinates 602 of the two people from camera server 225. According tothe camera server 225, the first person is at coordinate 602 (X₅, Y₅)and the second person is at coordinate 602 (X₆, Y₆). Central server 240also receives the physical coordinates 602 of the two people from LiDARserver 230 during the third time window 332C. According to the LiDARserver 230, the first person is at coordinate 602 (X₁₁, Y₁₁) and thesecond person is at coordinate 602 (X₁₂, Y₁₂) during the third timewindow 332C. Additionally, central server 240 learns from weight server235 that a particular weight sensor 500 detected that two items 130 weretaken during the third time window 332C.

In response to learning that a weight sensor 500 detected that two items130 were taken, central server 240 may undergo additional analysis todetermine which person took those two items 130. Central server 240performs any suitable process for determining which person took items130. Several of these processes are disclosed in U.S. application Ser.No. 16/663,710 entitled, “Topview Object Tracking Using a Sensor Array”(attorney docket no. (090278.0180), the contents of which areincorporated by reference herein.

FIG. 6B shows central server 240 performing an example analysis todetermine which person took items 130. As seen in FIG. 6B, centralserver 240 first determines the physical coordinates 602 of the twopeople during the third time window 332C. Central server 240 determinesthat the first person was at coordinate 602 (X₅, Y₅) during the thirdtime window 332C and the second person was at coordinate 602 (X₆, Y₆)during the third time window 332C. Central server 240 also determinesthe physical location of the weight sensor 500 that detected the itemsthat were taken. In example of FIG. 6B, central server 240 determinesthat the weight sensor 500 is located at coordinate 602 (X₁₃, Y₁₃).

Central server 240 then determines the distance from each person to theweight sensor 500. Central server 240 determines that the first personis a distance 1 from the weight sensor 500 and that the second person isa distance 2 from the weight sensor 500. Central server 240 thendetermines which person was closer to the weight sensor 500. In theexample of FIG. 4B, central server 240 determines that distance 1 isless than distance 2 and, thus, the first person was closer to theweight sensor 500 than the second person. As a result, central server240 determines that the first person took the two items 130 during thethird time window 332C and that the first person should be charged forthese two items 130.

FIG. 6C illustrates an example method 600 for operating central server240. In particular embodiments, central server 240 performs the steps ofmethod 600 to determine which person in a space took an item 130.

Central server 240 begins by receiving coordinates 602 of a first personin a space during a time window 332 in step 605. In step 610, centralserver 240 receives the coordinates 602 of a second person in the spaceduring the time window 332. Central server 240 receives an indicationthat an item 130 was taken during the time window 332 in step 615. Inresponse to reeving that indication, central server 240 analyzes theinformation to determine which person took that item 130.

In step 620, central server 240 determines that the first person wascloser to the item 130 than the second person during the time window332. Central server 240 may make this determination based on determineddistances between the people and a weight sensor 500 that detected thatthe item 130 was removed. In step 625, central server 240 determinesthat the first person took the item 130 during the time window 332 inresponse to determining that the first person was closer to the item 130than the second person. The first person may then be charged for theitem 130 when the first person exits the store 100.

Modifications, additions, or omissions may be made to method 600depicted in FIG. 6C. Method 600 may include more, fewer, or other steps.For example, steps may be performed in parallel or in any suitableorder. While discussed as central server 240 performing the steps, anysuitable component of tracking system 132 may perform one or more stepsof the method.

VI. Hardware

FIG. 7 illustrates an example computer 700 used in tracking system 132.

Generally, computer 700 can be used to implement components of trackingsystem 132. For example, computer 700 can be used to implement a cameraclient 220, a camera server 225, a LiDAR server 230, a weight server235, and/or a central server 240. As seen in FIG. 7, computer 700includes various hardware components, such as a processor 705, a memory710, a graphics processor 715, input/output ports 720, a communicationinterface 725, and a bus 730. This disclosure contemplates thecomponents of computer 700 being configured to perform any of thefunctions of camera client 220, camera server 225, LiDAR server 230,weight server 235, and/or central server 240 discussed herein. Circuitboard 565 may also include certain components of computer 700.

Processor 705 is any electronic circuitry, including, but not limited tomicroprocessors, application specific integrated circuits (ASIC),application specific instruction set processor (ASIP), and/or statemachines, that communicatively couples to memory 710 and controls theoperation of computer 700. Processor 705 may be 8-bit, 16-bit, 32-bit,64-bit or of any other suitable architecture. Processor 705 may includean arithmetic logic unit (ALU) for performing arithmetic and logicoperations, processor registers that supply operands to the ALU andstore the results of ALU operations, and a control unit that fetchesinstructions from memory and executes them by directing the coordinatedoperations of the ALU, registers and other components. Processor 705 mayinclude other hardware that operates software to control and processinformation. Processor 705 executes software stored on memory to performany of the functions described herein. Processor 705 controls theoperation and administration of computer 700 by processing informationreceived from memory 710 and/or other computers 700. Processor 705 maybe a programmable logic device, a microcontroller, a microprocessor, anysuitable processing device, or any suitable combination of thepreceding. Processor 705 is not limited to a single processing deviceand may encompass multiple processing devices.

Memory 710 may store, either permanently or temporarily, data,operational software, or other information for processor 705. Memory 710may include any one or a combination of volatile or non-volatile localor remote devices suitable for storing information. For example, memory710 may include random access memory (RAM), read only memory (ROM),magnetic storage devices, optical storage devices, or any other suitableinformation storage device or a combination of these devices. Thesoftware represents any suitable set of instructions, logic, or codeembodied in a computer-readable storage medium. For example, thesoftware may be embodied in memory 710, a disk, a CD, or a flash drive.In particular embodiments, the software may include an applicationexecutable by processor 705 to perform one or more of the functionsdescribed herein.

Graphics processor 715 may be any electronic circuitry, including, butnot limited to microprocessors, application specific integrated circuits(ASIC), application specific instruction set processor (ASIP), and/orstate machines, that receives and analyzes video data. For example,graphics processor 715 may process video data to determine the propersignals to send to a display so that the display displays an appropriateimage. Graphics processor 715 may also process video data to identifycertain characteristics (e.g., people or objects) within the video.Graphics processor 715 may be a component of a video card that isinstalled in computer 700.

Input/output ports 720 allow peripheral devices to connect to computer700. Ports 720 may be any suitable ports, such as, parallel ports,serial ports, optical ports, video ports, network ports, etc. Peripheraldevices such as keyboards, mouses, joysticks, optical tracking devices,trackpads, touchpads, etc. can connect to computer 700 through ports720. Input and output signals are communicated between computer 700 andthe peripheral devices through ports 720.

Communication interface 725 includes any suitable hardware and/orsoftware to communicate over a network. For example, communicationinterface 725 may include a mode, network card, ethernetport/controller, wireless radio/controller, cellular radio/controller,and/or universal serial bus port/controller. Computer 700 may usecommunication interface 725 to communicate with other devices over acommunication network.

Bus 730 allows components of computer 700 to communicate with oneanother. Computer 700 may include a bus controller 730 that managescommunication over bus 730.

As described above in the corresponding descriptions of FIGS. 3U-3Y, thepresent disclosure contemplates determining people detections includingone or more of contours, bounding areas/boxes, and segmentation masksassociated with people in frames. The corresponding description belowincludes a detailed explanation of how the people detections aredetermined and used in tracking those people throughout the store. Inthe present disclosure, bounding areas and bounding boxes are usedinterchangeably.

Auto-Exclusion Zones

Referring back to FIGS. 1A-1C, in order to track the movement of peoplein the space, the tracking system 132 should generally be able todistinguish between the people (i.e., the target objects) and otherobjects (i.e., non-target objects), such as the racks 115, displays, andany other non-human objects in the store 100. Otherwise, the trackingsystem 132 may waste memory and processing resources detecting andattempting to track these non-target objects. As described elsewhere inthis disclosure (e.g., in FIGS. 13-16 and the corresponding descriptionbelow), in some cases, people may be tracked may be performed bydetecting one or more contours in a set of image frames (e.g., a video)and monitoring movements of the contour between frames. A contour isgenerally a curve associated with an edge of a representation of aperson in an image. While the tracking system 132 may detect contours inorder to track people, in some instances, it may be difficult todistinguish between contours that correspond to people (e.g., or othertarget objects) and contours associated with non-target objects, such asracks 115, signs, product displays, and the like.

Even if cameras 305 are calibrated at installation to account for thepresence of non-target objects, in many cases, it may be challenging toreliably and efficiently recalibrate the cameras 305 to account forchanges in positions of non-target objects that should not be tracked inthe space. For example, if a rack 115, sign, product display, or otherfurniture or object in space is added, removed, or moved (e.g., allactivities which may occur frequently and which may occur withoutwarning and/or unintentionally), one or more of the cameras 305 mayrequire recalibration or adjustment. Without this recalibration oradjustment, it is difficult or impossible to reliably track people inthe space. Prior to this disclosure, there was a lack of tools forefficiently recalibrating and/or adjusting sensors, such as cameras 305,in a manner that would provide reliable tracking.

This disclosure encompasses the recognition not only of the previouslyunrecognized problems described above (e.g., with respect to trackingpeople in space, which may change over time) but also provides uniquesolutions to these problems. As described in this disclosure, during aninitial time period before people are tracked, pixel regions from eachcamera 305 may be determined that should be excluded during subsequenttracking. For example, during the initial time period, the space may notinclude any people such that contours detected by each camera 305correspond only to non-target objects in the space for which tracking isnot desired. Thus, pixel regions, or “auto-exclusion zones,”corresponding to portions of each image generated by cameras 305 thatare not used for object detection and tracking (e.g., the pixelcoordinates of contours that should not be tracked). For instance, theauto-exclusion zones may correspond to contours detected in images thatare associated with non-target objects, contours that are spuriouslydetected at the edges of a sensor's field-of-view, and the like).Auto-exclusion zones can be determined automatically at any desired orappropriate time interval to improve the usability and performance oftracking system 132.

After the auto-exclusion zones are determined, the tracking system 132may proceed to track people in the space. The auto-exclusion zones areused to limit the pixel regions used by each camera 305 for trackingpeople. For example, pixels corresponding to auto-exclusion zones may beignored by the tracking system 132 during tracking. In some cases, adetected person (e.g., or other target object) may be near or partiallyoverlapping with one or more auto-exclusion zones. In these cases, thetracking system 132 may determine, based on the extent to which apotential target object's position overlaps with the auto-exclusionzone, whether the target object will be tracked. This may reduce oreliminate false positive detection of non-target objects during persontracking in the space, while also improving the efficiency of trackingsystem 132 by reducing wasted processing resources that would otherwisebe expended attempting to track non-target objects. In some embodiments,a map of the space may be generated that presents the physical regionsthat are excluded during tracking (i.e., a map that presents arepresentation of the auto-exclusion zone(s) in the physical coordinatesof the space). Such a map, for example, may facilitate trouble-shootingof the tracking system by allowing an administrator to visually confirmthat people can be tracked in appropriate portions of the space.

FIG. 8 illustrates the determination of auto-exclusion zones 1110, 1114and the subsequent use of these auto-exclusion zones 1110, 1114 forimproved tracking of people (e.g., or other target objects) in thespace. In general, during an initial time period (t<t₀), top-view imageframes are received by the client(s) 220 and/or camera server 225 fromcameras 305 and used to determine auto-exclusion zones 1110, 1114. Forinstance, the initial time period at t<t₀ may correspond to a time whenno people are in the space. For example, if the space is open to thepublic during a portion of the day, the initial time period may bebefore the space is opened to the public. In some embodiments, thecamera server 225 and/or client 220 may provide, for example, an alertor transmit a signal indicating that the space should be emptied ofpeople (e.g., or other target objects to be tracked) in order forauto-exclusion zones 1110, 1114 to be identified. In some embodiments, auser may input a command (e.g., via any appropriate interface coupled tothe camera server 225 and/or client(s) 220) to initiate thedetermination of auto-exclusion zones 1110, 1114 immediately or at oneor more desired times in the future (e.g., based on a schedule).

An example top-view image frame 1102 used for determining auto-exclusionzones 1110, 1114 is shown in FIG. 8. Image frame 1102 includes arepresentation of a first object 1104 (e.g., a rack 115) and arepresentation of a second object 1106. For instance, the first object1104 may be a rack 115, and the second object 1106 may be a productdisplay or any other non-target object in the space. In someembodiments, the second object 1106 may not correspond to an actualobject in the space but may instead be detected anomalously because oflighting in the space and/or a sensor error. Each camera 305 generallygenerates at least one frame 1102 during the initial time period, andthese frame(s) 1102 is/are used to determine correspondingauto-exclusion zones 1110, 1114 for the camera 305. For instance, thecamera client 220 may receive the top-view image 1102, and detectcontours (i.e., the dashed lines around zones 1110, 1114) correspondingto the auto-exclusion zones 1110, 1114 as illustrated in view 1108. Thecontours of auto-exclusion zones 1110, 1114 generally correspond tocurves that extend along a boundary (e.g., the edge) of objects 1104,1106 in image 1102. The view 1108 generally corresponds to apresentation of image 1102 in which the detected contours correspondingto auto-exclusion zones 1110, 1114 are presented but the correspondingobjects 1104, 1106, respectively, are not shown. For an image frame 1102that includes color and depth data, contours for auto-exclusion zones1110, 1114 may be determined at a given depth (e.g., a distance awayfrom camera 305) based on the color data in the image 1102. For example,a steep gradient of a color value may correspond to an edge of an objectand used to determine, or detect, a contour. For example, contours forthe auto-exclusion zones 1110, 1114 may be determined using any suitablecontour or edge detection method such as Canny edge detection,threshold-based detection, or the like.

The client 220 determines pixel coordinates 1112 and 1116 correspondingto the locations of the auto-exclusions zones 1110 and 1114,respectively. The pixel coordinates 1112, 1116 generally correspond tothe locations (e.g., row and column numbers) in the image frame 1102that should be excluded during tracking. In general, objects associatedwith the pixel coordinates 1112, 1116 are not tracked by the trackingsystem 132. Moreover, certain objects which are detected outside of theauto-exclusion zones 1110, 1114 may not be tracked under certainconditions. For instance, if the position of the object (e.g., theposition associated with region 1120, discussed below with respect toview 1114) overlaps at least a threshold amount with an auto-exclusionzone 1110, 1114, the object may not be tracked. This prevents thetracking system 132 (i.e., or the local client 220 associated with acamera 305 or a subset of cameras 305) from attempting to unnecessarilytrack non-target objects. In some cases, auto-exclusion zones 1110, 1114correspond to non-target (e.g., inanimate) objects in the field-of-viewof a camera 305 (e.g., a rack 115, which is associated with contour1110). However, auto-exclusion zones 1110, 1114 may also oralternatively correspond to other aberrant features or contours detectedby a camera 305 (e.g., caused by sensor errors, inconsistent lighting,or the like).

Following the determination of pixel coordinates 1112, 1116 to excludeduring tracking, objects may be tracked during a subsequent time periodcorresponding to t>t₀. An example image frame 1118 generated duringtracking is shown in FIG. 8. In frame 1118, region 1120 is detected aspossibly corresponding to what may or may not be a target object. Forexample, region 1120 may correspond to a pixel mask or bounding boxgenerated based on a contour detected in frame 1102. For example, apixel mask may be generated to fill in the area inside the contour or abounding box may be generated to encompass the contour. For example, apixel mask may include the pixel coordinates within the correspondingcontour. For instance, the pixel coordinates 1112 of auto-exclusion zone1110 may effectively correspond to a mask that overlays or “fills in”the auto-exclusion zone 1110. Following the detection of region 1120,the client 220 determines whether the region 1120 corresponds to atarget object which should tracked or is sufficiently overlapping withauto-exclusion zone 1114 to consider region 1120 as being associatedwith a non-target object. For example, the client 220 may determinewhether at least a threshold percentage of the pixel coordinates 1116overlap with (e.g., are the same as) pixel coordinates of region 1120.The overlapping region 1122 of these pixel coordinates is illustrated inframe 1118. For example, the threshold percentage may be about 50% ormore. In some embodiments, the threshold percentage may be as small asabout 10%. In response to determining that at least the thresholdpercentage of pixel coordinates overlap, the client 220 generally doesnot determine a pixel position for tracking the object associated withregion 1120. However, if overlap 1122 correspond to less than thethreshold percentage, an object associated with region 1120 is tracked,as described further below (e.g., with respect to FIGS. 13-16).

As described above, cameras 305 may be arranged such that adjacentcameras 305 have overlapping fields-of-view. For instance,fields-of-view of adjacent cameras 305 may overlap by between about 10%to 30%. As such, the same object may be detected by two differentcameras 305 and either included or excluded from tracking in the imageframes received from each camera 305 based on the unique auto-exclusionzones determined for each camera 305. This may facilitate more reliabletracking than was previously possible, even when one camera 305 may havea large auto-exclusion zone (i.e., where a large proportion of pixelcoordinates in image frames generated by the camera 305 are excludedfrom tracking). Accordingly, if one camera 305 malfunctions, adjacentcameras 305 may still provide adequate tracking in the space.

If region 1120 corresponds to a target object (i.e., a person to trackin the space), the tracking system 132 proceeds to track the region1120. Example methods of tracking are described in greater detail belowwith respect to FIGS. 13-16. In some embodiments, the camera server 225uses the pixel coordinates 1112, 1116 to determine correspondingphysical coordinates (e.g., coordinates 1212, 1216 illustrated in FIG.9, described below). For instance, the client 220 may determine pixelcoordinates 1112, 1116 corresponding to the local auto-exclusion zones1110, 1114 of a camera 305 and transmit these coordinates 1112, 1116 tothe camera server 235. As shown in FIG. 9, the camera server 225 may usethe pixel coordinates 1112, 1116 received from the camera 305 todetermine corresponding physical coordinates 1210, 1216. For instance, ahomography generated for each camera 305 (see FIGS. 3A-3T and thecorresponding description above), which associates pixel coordinates(e.g., coordinates 1112, 1116) in an image generated by a given camera305 to corresponding physical coordinates (e.g., coordinates 1212, 1216)in the space, may be employed to convert the excluded pixel coordinates1112, 1116 (of FIG. 8) to excluded physical coordinates 1212, 1216 inthe space. These excluded coordinates 1210, 1216 may be used along withother coordinates from other cameras 305 to generate the globalauto-exclusion zone map 1200 of the space which is illustrated in FIG.9. This map 1200, for example, may facilitate trouble-shooting of thetracking system 132 by facilitating quantification, identification,and/or verification of physical regions 1202 of space where objects may(and may not) be tracked. This may allow an administrator or otherindividual to visually confirm that objects can be tracked inappropriate portions of the space). If regions 1202 correspond to knownhigh-traffic zones of the space, system maintenance may be appropriate(e.g., which may involve replacing, adjusting, and/or adding additionalcameras 305).

FIG. 10 is a flowchart illustrating an example method 1300 forgenerating and using auto-exclusion zones (e.g., zones 1110, 1114 ofFIG. 8). Method 1300 may begin at step 1302 where one or more imageframes 1102 are received during an initial time period. As describedabove, the initial time period may correspond to an interval of timewhen no person is moving throughout the space, or when no person iswithin the field-of-view of one or more cameras 305 from which the imageframe(s) 1102 is/are received. In a typical embodiment, one or moreimage frames 1102 are generally received from each camera 305 of thetracking system 132, such that local regions (e.g., auto-exclusion zones1110, 1114) to exclude for each camera 305 may be determined. In someembodiments, a single image frame 1102 is received from each camera 305to detect auto-exclusion zones 1110, 1114. However, in otherembodiments, multiple image frames 1102 are received from each camera305. Using multiple image frames 1102 to identify auto-exclusions zones1110, 1114 for each camera 305 may improve the detection of any spuriouscontours or other aberrations that correspond to pixel coordinates(e.g., coordinates 1112, 1116 of FIG. 8) which should be ignored orexcluded during tracking.

At step 1304, contours (e.g., dashed contour lines corresponding toauto-exclusion zones 1110, 1114 of FIG. 8) are detected in the one ormore image frames 1102 received at step 1302. Any appropriate contourdetection algorithm may be used including but not limited to those basedon Canny edge detection, threshold-based detection, and the like. Insome embodiments, the unique contour detection approaches described inthis disclosure may be used (e.g., to distinguish closely spacedcontours in the field-of-view, as described below, for example, withrespect to FIGS. 11 and 12). At step 1306, pixel coordinates (e.g.,coordinates 1112, 1116 of FIG. 8) are determined for the detectedcontours (from step 1304). The coordinates may be determined, forexample, based on a pixel mask that overlays the detected contours. Apixel mask may for example, correspond to pixels within the contours. Insome embodiments, pixel coordinates correspond to the pixel coordinateswithin a bounding box determined for the contour (e.g., as illustratedin FIG. 11, described below). For instance, the bounding box may be arectangular box with an area that encompasses the detected contour. Atstep 1308, the pixel coordinates are stored. For instance, the client220 may store the pixel coordinates corresponding to auto-exclusionzones 1110, 1114 in memory (e.g., memory 710 of FIG. 7, describedabove). As described above, the pixel coordinates may also oralternatively be transmitted to the camera server 225 (e.g., to generatea map 1200 of the space, as illustrated in the example of FIG. 9).

At step 1310, the client 220 receives an image frame 1118 during asubsequent time during which tracking is performed (i.e., after thepixel coordinates corresponding to auto-exclusion zones are stored atstep 1308). The frame is received from camera 305 and includes arepresentation of an object in the space. At step 1312, a contour isdetected in the frame received at step 1310. For example, the contourmay correspond to a curve along the edge of object represented in theframe 1102. The pixel coordinates determined at step 1306 may beexcluded (or not used) during contour detection. For instance, imagedata may be ignored and/or removed (e.g., given a value of zero, or thecolor equivalent) at the pixel coordinates determined at step 1306, suchthat no contours are detected at these coordinates. In some cases, acontour may be detected outside of these coordinates. In some cases, acontour may be detected that is partially outside of these coordinatesbut overlaps partially with the coordinates (e.g., as illustrated inimage 1118 of FIG. 8).

At step 1314, the client 220 generally determines whether the detectedcontour has a pixel position that sufficiently overlaps with pixelcoordinates of the auto-exclusion zones 1110, 1114 determined at step1306. If the coordinates sufficiently overlap, the contour or region1120 (i.e., and the associated object) is not tracked in the frame. Forinstance, as described above, the client 220 may determine whether thedetected contour or region 1120 overlaps at least a threshold percentage(e.g., of 50%) with a region associated with the pixel coordinates(e.g., see overlapping region 1122 of FIG. 8). If the criteria of step1314 are satisfied, the client 220 generally, at step 1316, does notdetermine a pixel position for the contour detected at step 1312. Assuch, no pixel position is reported to the camera server 235, therebyreducing or eliminating the waste of processing resources associatedwith attempting to track an object when it is not a target object forwhich tracking is desired.

Otherwise, if the criteria of step 1314 are satisfied, the client 220determines a pixel position for the contour or region 1120 at step 1318.Determining a pixel position from a contour may involve, for example,(i) determining a region 1120 (e.g., a pixel mask or bounding box)associated with the contour and (ii) determining a centroid or othercharacteristic position of the region as the pixel position. At step1320, the determined pixel position is transmitted to the camera server225 to facilitate global tracking, for example, using predeterminedhomographies, as described elsewhere in this disclosure (e.g., withrespect to FIGS. 13-16). For example, the camera server 225 may receivethe determined pixel position, access a homography associating pixelcoordinates in images generated by the camera 305 from which the frameat step 1310 was received to physical coordinates in the space, andapply the homography to the pixel coordinates to generate correspondingphysical coordinates for the tracked object associated with the contourdetected at step 1312.

Modifications, additions, or omissions may be made to method 1300depicted in FIG. 10. Method 1300 may include more, fewer, or othersteps. For example, steps may be performed in parallel or in anysuitable order. While at times discussed as tracking system 132,client(s) 220, camera server 235, or components of any of thereofperforming steps, any suitable system or components of the system mayperform one or more steps of the method.

Contour-Based Detection of Closely Spaced People

In some cases, two people are near each other, making it difficult orimpossible to reliably detect and/or track each person (e.g., or othertarget object) using conventional tools. In some cases, the people maybe initially detected and tracked using depth images at an approximatewaist depth (i.e., a depth corresponding to the waist height of anaverage person being tracked). Tracking at an approximate waist depthmay be more effective at capturing all people regardless of their heightor mode of movement. For instance, by detecting and tacking people at anapproximate waist depth, the tracking system 132 is highly likely todetect tall and short individuals and individuals who may be usingalternative methods of movement (e.g., wheelchairs, and the like).However, if two people with a similar height are standing near eachother, it may be difficult to distinguish between the two people in thetop-view images at the approximate waist depth. Rather than detectingtwo separate people, the tracking system 132 may initially detect thepeople as a single larger object.

This disclosure encompasses the recognition that at a decreased depth(i.e., a depth nearer the heads of the people), the people may be morereadily distinguished. This is because the people's heads are morelikely to be imaged at the decreased depth, and their heads are smallerand less likely to be detected as a single merged region (or contour, asdescribed in greater detail below). As another example, if two peopleenter the space standing close to one another (e.g., holding hands),they may appear to be a single larger object. Since the tracking system132 may initially detect the two people as one person, it may bedifficult to properly identify these people if these people separatewhile in the space. As yet another example, if two people who brieflystand close together are momentarily “lost” or detected as only asingle, larger object, it may be difficult to correctly identify thepeople after they separate from one another.

As described elsewhere in this disclosure (e.g., with respect to FIGS.8-10 and 13-16), people (e.g., the people in the example scenariosdescribed above) may be tracked by detecting contours in top-view imageframes generated by cameras 305 and tracking the positions of thesecontours. However, when two people are closely spaced, a single mergedcontour (see merged contour 1420 of FIG. 11 described below) may bedetected in a top-view image of the people. This single contourgenerally cannot be used to track each person individually, resulting inconsiderable downstream errors during tracking. For example, even if twopeople separate after having been closely spaced, it may be difficult orimpossible using previous tools to determine which person was which, andthe identity of each person may be unknown after the two peopleseparate. Prior to this disclosure, there was a lack of reliable toolsfor detecting people (e.g., and other target objects) under the examplescenarios described above and under other similar circumstances.

The systems and methods described in this disclosure provideimprovements to previous technology by facilitating the improveddetection of closely spaced people. For example, the systems and methodsdescribed in this disclosure may facilitate the detection of individualpeople when contours associated with these people would otherwise bemerged, resulting in the detection of a single person using conventionaldetection strategies. In some embodiments, improved contour detection isachieved by detecting contours at different depths (e.g., at least twodepths) to identify separate contours at a second depth within a largermerged contour detected at a first depth used for tracking. For example,if two people are standing near each other such that contours are mergedto form a single contour, separate contours associated with heads of thetwo closely spaced people may be detected at a depth associated with thepersons' heads. In some embodiments, a unique statistical approach maybe used to differentiate between the two people by selecting boundingregions for the detected contours with a low similarity value. In someembodiments, certain criteria are satisfied to ensure that the detectedcontours correspond to separate people, thereby providing more reliableperson (e.g., or other target object) detection than was previouslypossible. For example, two contours detected at an approximate headdepth may be required to be within a threshold size range in order forthe contours to be used for subsequent tracking. In some embodiments, anartificial neural network may be employed to detect separate people thatare closely spaced by analyzing top-view images at different depths.

FIG. 11 is a diagram illustrating the detection of two closely spacedpeople 1402, 1404 based on top-view depth images 1412 and angled-viewimages 1414 received from cameras 305 a,b using the tracking system 132.In one embodiment, cameras 305 a,b may each be one of cameras 305 oftracking system 132 described above with respect to FIG. 2B. In anotherembodiment, cameras 305 a,b may each be one of cameras 305 of a separatevirtual store system (e.g, layout cameras and/or rack cameras) asdescribed in U.S. patent application Ser. No. 16/664,470 entitled,“Customer-Based Video Feed” (attorney docket no. 090278.0187) which isincorporated by reference herein. In this embodiment, the cameras 305 oftracking system 132 may be mapped to the cameras 305 of the virtualstore system using a homography. Moreover, this embodiment can retrieveidentifiers and the relative position of each person from the cameras305 of the virtual store system using the homography between trackingsystem 132 and the virtual store system. Generally, camera 305 a is anoverhead sensor configured to generate top-view depth images 1412 (e.g.,color and/or depth images) of at least a portion of the space. Camera305 a may be mounted, for example, in a ceiling of the space. Camera 305a may generate image data corresponding to a plurality of depths whichinclude but are not necessarily limited to the depths 1410 a-cillustrated in FIG. 11. Depths 1410 a-c are generally distances measuredfrom the camera 305 a. Each depth 1410 a-c may be associated with acorresponding height (e.g., from the floor of the space in which people1402, 1404 are detected and/or tracked). Camera 305 a observes afield-of-view 1408 a. Top-view images 1412 generated by camera 305 a maybe transmitted to the camera client 220 a. The camera client 220 a iscommunicatively coupled (e.g., via wired connection of wirelessly) tothe camera 305 a and the camera server 235. Camera server 225 isdescribed above with respect to FIG. 2B.

In this example, camera 305 b is an angled-view sensor, which isconfigured to generate angled-view images 1414 (e.g., color and/or depthimages) of at least a portion of the space. Camera 305 b has a field ofview 1408 b, which overlaps with at least a portion of the field-of-view1408 a of camera 305 a. The angled-view images 1414 generated by theangled-view camera 305 b are transmitted to camera client 220 b. Cameraclient 220 b may be a client 220 described above with respect to FIG. 2.In the example of FIG. 11, cameras 305 a,b are coupled to differentcamera clients 220 a,b. However, it should be understood that the samecamera client 220 may be used for both cameras 305 a,b (e.g., such thatclients 220 a,b are the same client 220). In some cases, the use ofdifferent camera clients 220 a,b for cameras 305 a,b may provideimproved performance because image data may still be obtained for thearea shared by fields-of-view 1408 a,b even if one of the clients 220a,b were to fail.

In the example scenario illustrated in FIG. 11, people 1402, 1404 arelocated sufficiently close together such that conventional objectdetection tools fail to detect the individual people 1402, 1404 (e.g.,such that people 1402, 1404 would not have been detected as separateobjects). This situation may correspond, for example, to the distance1406 a between people 1402, 1404 being less than a threshold distance1406 b (e.g., of about 6 inches). The threshold distance 1406 b cangenerally be any appropriate distance determined for the system 132. Forexample, the threshold distance 1406 b may be determined based onseveral characteristics of the system 1400 and the people 1402, 1404being detected. For example, the threshold distance 1406 b may be basedon one or more of the distance of the camera 305 a from the people 1402,1404, the size of the people 1402, 1404, the size of the field-of-view1408 a, the sensitivity of the camera 305 a, and the like. Accordingly,the threshold distance 1406 b may range from just over zero inches toover six inches depending on these and other characteristics of thetracking system 132. People 1402, 1404 may be any target object anindividual may desire to detect and/or track based on data (i.e.,top-view images 1412 and/or angled-view images 1414) from cameras 305a,b.

The camera client 220 a detects contours in top-view images 1412received from camera 305 a. Typically, the camera client 220 a detectscontours at an initial depth 1410 a. The initial depth 1410 a may beassociated with, for example, a predetermined height (e.g., from theground) which has been established to detect and/or track people 1402,1404 through the space. For example, for tracking humans, the initialdepth 1410 a may be associated with an average shoulder or waist heightof people expected to be moving in the space (e.g., a depth which islikely to capture a representation for both tall and short peopletraversing the space). The camera client 220 a may use the top-viewimages 1412 generated by camera 305 a to identify the top-view image1412 corresponding to when a first contour 1402 a associated with thefirst person 1402 merges with a second contour 1404 a associated withthe second person 1404. View 1416 illustrates contours 1402 a, 1404 a ata time prior to when these contours 1402 a, 1404 a merge (i.e., prior toa time (t_(close)) when the first and second people 1402, 1404 arewithin the threshold distance 1406 b of each other). View 1416corresponds to a view of the contours detected in a top-view image 1412received from camera 305 a (e.g., with other objects in the image notshown).

A subsequent view 1418 corresponds to the image 1412 at or neart_(close) when the people 1402, 1404 are closely spaced and the firstand second contours 1402 a, 1404 a merge to form merged contour 1420.The camera client 220 a may determine a region 1422 which corresponds toa “size” of the merged contour 1420 in image coordinates (e.g., a numberof pixels associated with contour 1420). For example, region 1422 maycorrespond to a pixel mask or a bounding box determined for contour1420. Example approaches to determining pixel masks and bounding boxesare described above with respect to step 1304 of FIG. 10. For example,region 1422 may be a bounding box determined for the contour 1420 usinga non-maximum suppression object-detection algorithm. For instance, thecamera client 220 a may determine a plurality of bounding boxesassociated with the contour 1420. For each bounding box, the client 220a may calculate a score. The score, for example, may represent an extentto which that bounding box is similar to the other bounding boxes. Thecamera client 220 a may identify a subset of the bounding boxes with ascore that is greater than a threshold value (e.g., 80% or more), anddetermine region 1422 based on this identified subset. For example,region 1422 may be the bounding box with the highest score or a boundingcomprising regions shared by bounding boxes with a score that is abovethe threshold value.

In order to detect the individual people 1402 and 1404, the cameraclient 220 a may access images 1412 at a decreased depth (i.e., at oneor both of depths 1412 b and 1412 c) and use this data to detectseparate contours 1402 b, 1404 b, illustrated in view 1424. In otherwords, the camera client 220 a may analyze the images 1412 at a depthnearer the heads of people 1402, 1404 in the images 1412 in order todetect the separate people 1402, 1404. In some embodiments, thedecreased depth may correspond to an average or predetermined headheight of persons expected to be detected by the tracking system 132 inthe space. In some cases, contours 1402 b, 1404 b may be detected at thedecreased depth for both people 1402, 1404.

However, in other cases, the camera client 220 a may not detect bothheads at the decreased depth. For example, if a child and an adult areclosely spaced, only the adult's head may be detected at the decreaseddepth (e.g., at depth 1410 b). In this scenario, the camera client 220 amay proceed to a slightly increased depth (e.g., to depth 1410 c) todetect the head of the child. For instance, in such scenarios, thecamera client 220 a iteratively increases the depth from the decreaseddepth towards the initial depth 1410 a in order to detect two distinctcontours 1402 b, 1404 b (e.g., for both the adult and the child in theexample described above). For instance, the depth may first be decreasedto depth 1410 b and then increased to depth 1410 c if both contours 1402b and 1404 b are not detected at depth 1410 b. This iterative process isdescribed in greater detail below with respect to method 1500 of FIG.12.

As described elsewhere in this disclosure, in some cases, the trackingsystem 132 may maintain a record of features, or descriptors, associatedwith each tracked person (see, e.g., FIG. 19, described below). As such,the camera client 220 a may access this record to determine uniquedepths that are associated with the people 1402, 1404, which are likelyassociated with merged contour 1420. For instance, depth 1410 b may beassociated with a known head height of person 1402, and depth 1412 c maybe associated with a known head height of person 1404.

Once contours 1402 b and 1404 b are detected, the camera clientdetermines a region 1402 c associated with pixel coordinates 1402 d ofcontour 1402 b and a region 1404 c associated with pixel coordinates1404 d of contour 1404 b. For example, as described above with respectto region 1422, regions 1402 c and 1404 c may correspond to pixel masksor bounding boxes generated based on the corresponding contours 1402 b,1404 b, respectively. For example, pixel masks may be generated to “fillin” the area inside the contours 1402 b, 1404 b or bounding boxes may begenerated which encompass the contours 1402 b, 1404 b. The pixelcoordinates 1402 d, 1404 d generally correspond to the set of positions(e.g., rows and columns) of pixels within regions 1402 c, 1404 c.

In some embodiments, a unique approach is employed to more reliablydistinguish between closely spaced people 1402 and 1404 and determineassociated regions 1402 c and 1404 c. In these embodiments, the regions1402 c and 1404 c are determined using a unique method referred to inthis disclosure as “non-minimum suppression.” Non-minimum suppressionmay involve, for example, determining bounding boxes associated with thecontour 1402 b, 1404 b (e.g., using any appropriate object detectionalgorithm as appreciated by a person of skilled in the relevant art).For each bounding box, a score may be calculated. As described abovewith respect to non-maximum suppression, the score may represent anextent to which the bounding box is similar to the other bounding boxes.However, rather than identifying bounding boxes with high scores (e.g.,as with non-maximum suppression), a subset of the bounding boxes isidentified with scores that are less than a threshold value (e.g., ofabout 20%). This subset may be used to determine regions 1402 c, 1404 c.For example, regions 1402 c, 1404 c may include regions shared by eachbounding box of the identified subsets. In other words, bounding boxesthat are not below the minimum score are “suppressed” and not used toidentify regions 1402 b, 1404 b.

Prior to assigning a position or identity to the contours 1402 b, 1404 band/or the associated regions 1402 c, 1404 c, the camera client 220 amay first check whether criteria are satisfied for distinguishing theregion 1402 c from region 1404 c. The criteria are generally designed toensure that the contours 1402 b, 1404 b (and/or the associated regions1402 c, 1404 c) are appropriately sized, shaped, and positioned to beassociated with the heads of the corresponding people 1402, 1404. Thesecriteria may include one or more requirements. For example, onerequirement may be that the regions 1402 c, 1404 c overlap by less thanor equal to a threshold amount (e.g., of about 50%, e.g., of about 10%).Generally, the separate heads of different people 1402, 1404 should notoverlap in a top-view image 1412. Another requirement may be that theregions 1402 c, 1404 c are within (e.g., bounded by, e.g., encompassedby) the merged-contour region 1422. This requirement, for example,ensures that the head contours 1402 b, 1404 b are appropriatelypositioned above the merged contour 1420 to correspond to heads ofpeople 1402, 1404. If the contours 1402 b, 1404 b detected at thedecreased depth are not within the merged contour 1420, then thesecontours 1402 b, 1404 b are likely not the associated with heads of thepeople 1402, 1404 associated with the merged contour 1420.

Generally, if the criteria are satisfied, the camera client 220 aassociates region 1402 c with a first pixel position 1402 e of person1402 and associates region 1404 c with a second pixel position 1404 e ofperson 1404. Each of the first and second pixel positions 1402 e, 1404 egenerally corresponds to a single pixel position (e.g., row and column)associated with the location of the corresponding contour 1402 b, 1404 bin the image 1412. The first and second pixel positions 1402 e, 1404 eare included in the pixel positions 1426 which may be transmitted to thecamera server 225 to determine corresponding physical (e.g., global)positions 1428, for example, based on homographies 1430 (e.g., using apreviously determined homography for camera 305 a associating pixelcoordinates in images 1412 generated by camera 305 a to physicalcoordinates in the space).

As described above, camera 305 b is positioned and configured togenerate angled-view images 1414 of at least a portion of the fieldof-of-view 1408 a of camera 305 a. The camera client 220 b receives theangled-view images 1414 from the second camera 305 b. Because of itsdifferent (e.g., angled) view of people 1402, 1404 in the space, anangled-view image 1414 obtained at t_(close) may be sufficient todistinguish between the people 1402, 1404. A view 1432 of contours 1402d, 1404 d detected at t_(close) is shown in FIG. 11. The camera client220 b detects a contour 1402 f corresponding to the first person 1402and determines a corresponding region 1402 g associated with pixelcoordinates 1402 h of contour 1402 f The camera client 220 b detects acontour 1404 f corresponding to the second person 1404 and determines acorresponding region 1404 g associated with pixel coordinates 1404 h ofcontour 1404 f Since contours 1402 f, 1404 f do not merge and regions1402 g, 1404 g are sufficiently separated (e.g., they do not overlapand/or are at least a minimum pixel distance apart), the camera client220 b may associate region 1402 g with a first pixel position 1402 i ofthe first person 1402 and region 1404 g with a second pixel position1404 i of the second person 1404. Each of the first and second pixelpositions 1402 i, 1404 i generally corresponds to a single pixelposition (e.g., row and column) associated with the location of thecorresponding contour 1402 f, 1404 f in the image 1414. Pixel positions1402 i, 1404 i may be included in pixel positions 1434 which may betransmitted to camera server 225 to determine physical positions 1428 ofthe people 1402, 1404 (e.g., using a previously determined homographyfor camera 305 b associating pixel coordinates of images 1414 generatedby camera 305 b to physical coordinates in the space).

In an example operation of the tracking system 132 camera 305 a isconfigured to generate top-view color-depth images of at least a portionof the space. When people 1402 and 1404 are within a threshold distanceof each another, the camera client 220 a identifies an image frame(e.g., associated with view 1418) corresponding to a time stamp (e.g.,t_(close)) where contours 1402 a, 1404 a associated with the first andsecond person 1402, 1404, respectively, are merged and form contour1420. In order to detect each person 1402 and 1404 in the identifiedimage frame (e.g., associated with view 1418), the client 220 a mayfirst attempt to detect separate contours for each person 1402, 1404 ata first decreased depth 1410 b. As described above, depth 1410 b may bea predetermined height associated with an expected head height of peoplemoving through the space. In some embodiments, depth 1410 b may be adepth previously determined based on a measured height of person 1402and/or a measured height of person 1404. For example, depth 1410 b maybe based on an average height of the two people 1402, 1404. As anotherexample, depth 1410 b may be a depth corresponding to a predeterminedhead height of person 1402 (as illustrated in the example of FIG. 11).If two contours 1402 b, 1404 b are detected at depth 1410 b, thesecontours may be used to determine pixel positions 1402 e, 1404 e ofpeople 1402 and 1404, as described above.

If only one contour 1402 b is detected at depth 1410 b (e.g., if onlyone person 1402, 1404 is tall enough to be detected at depth 1410 b),the region associated with this contour 1402 b may be used to determinethe pixel position 1402 e of the corresponding person, and the nextperson may be detected at an increased depth 1410 c. Depth 1410 c isgenerally greater than 1410 b but less than depth 1410 a. In theillustrative example of FIG. 11, depth 1410 c corresponds to apredetermined head height of person 1404. If contour 1404 b is detectedfor person 1404 at depth 1410 c, a pixel position 1404 e is determinedbased on pixel coordinates 1404 d associated with the contour 1404 b(e.g., following determination that the criteria described above aresatisfied). If a contour 1404 b is not detected at depth 1410 c, theclient 220 a may attempt to detect contours at progressively increaseddepths until a contour is detected or a maximum depth (e.g., the initialdepth 1410 a) is reached. For example, the camera client 220 a maycontinue to search for the contour 1404 b at increased depths (i.e.,depths between depth 1410 c and the initial depth 1410 a). If themaximum depth (e.g., depth 1410 a) is reached without the contour 1404 bbeing detected, the client 220 a generally determines that the separatepeople 1402, 1404 cannot be detected.

FIG. 12 is a flowchart illustrating a method 1500 of operating trackingsystem 132 to detect closely spaced people 1402, 1404. Method 1500 maybegin at step 1502 where the camera client 220 a receives one or moreframes of top-view depth images 1412 generated by camera 305 a. At step1504, the camera client 220 a identifies a frame in which a firstcontour 1402 a associated with the first person 1402 is merged with asecond contour 1404 a associated with the second person 1404. Generally,the merged first and second contours (i.e., merged contour 1420) isdetermined at the first depth 1412 a in the depth images 1412 receivedat step 1502. The first depth 1412 a may correspond to a waist or shoulddepth of persons expected to be tracked in the space. The detection ofmerged contour 1420 corresponds to the first person 1402 being locatedin the space within a threshold distance 1406 b from the second person1404, as described above.

At step 1506, the camera client 220 a determines a merged-contour region1422. Region 1422 is associated with pixel coordinates of the mergedcontour 1420. For instance, region 1422 may correspond to coordinates ofa pixel mask that overlays the detected contour. As another example,region 1422 may correspond to pixel coordinates of a bounding boxdetermined for the contour (e.g., using any appropriate object detectionalgorithm). In some embodiments, a method involving non-maximumsuppression is used to detect region 1422. In some embodiments, region1422 is determined using an artificial neural network. For example, anartificial neural network may be trained to detect contours at variousdepths in top-view images generated by camera 305 a.

At step 1508, the depth at which contours are detected in the identifiedimage frame from step 1504 is decreased (e.g., to depth 1410 billustrated in FIG. 11). At step 1510 a, the camera client 220 adetermines whether a first contour (e.g., contour 1402 b) is detected atthe current depth. If the contour 1402 b is not detected, the cameraclient 220 a proceeds, at step 1512 a, to an increased depth (e.g., todepth 1410 c). If the increased depth corresponds to having reached amaximum depth (e.g., to reaching the initial depth 1410 a), the processends because the first contour 1402 b was not detected. If the maximumdepth has not been reached, the camera client 220 a returns to step 1510a and determines if the first contour 1402 b is detected at the newlyincreased current depth. If the first contour 1402 b is detected at step1510 a, the camera client 220 a, at step 1516 a, determines a firstregion 1402 c associated with pixel coordinates 1402 d of the detectedcontour 1402 b. In some embodiments, region 1402 c may be determinedusing a method of non-minimal suppression, as described above. In someembodiments, region 1402 c may be determined using an artificial neuralnetwork.

The same or a similar approach—illustrated in steps 1410 b, 1412 b, 1414b, and 1416 b—may be used to determine a second region 1404 c associatedwith pixel coordinates 1404 d of the contour 1404 b. For example, atstep 1510 b, the camera client 220 a determines whether a second contour1404 b is detected at the current depth. If the contour 1404 b is notdetected, the camera client 220 a proceeds, at step 1512 b, to anincreased depth (e.g., to depth 1410 c). If the increased depthcorresponds to having reached a maximum depth (e.g., to reaching theinitial depth 1410 a), the process ends because the second contour 1404b was not detected. If the maximum depth has not been reached, thecamera client 220 a returns to step 1510 b and determines if the secondcontour 1404 b is detected at the newly increased current depth. If thesecond contour 1404 b is detected at step 1410 a, the camera client 220a, at step 1516 a, determines a second region 1404 c associated withpixel coordinates 1404 d of the detected contour 1404 b. In someembodiments, region 1404 c may be determined using a method ofnon-minimal suppression or an artificial neural network, as describedabove.

At step 1518, the camera client 220 a determines whether criteria aresatisfied for distinguishing the first and second regions determined insteps 1516 a and 1516 b, respectively. For example, the criteria mayinclude one or more requirements. For example, one requirement may bethat the regions 1402 c, 1404 c overlap by less than or equal to athreshold amount (e.g., of about 10%). Another requirement may be thatthe regions 1402 c, 1404 c are within (e.g., bounded by, e.g.,encompassed by) the merged-contour region 1422 (determined at step1506). If the criteria are not satisfied, method 1500 generally ends.

Otherwise, if the criteria are satisfied at step 1518, the method 1500proceeds to steps 1520 and 1522 where the camera client 220 a associatesthe first region 1402 b with a first pixel position 1402 e of the firstperson 1402 (step 1520) and associates the second region 1404 b with afirst pixel position 1402 e of the first person 1404 (step 1522).Associating the regions 1402 c, 1404 c to pixel positions 1402 e, 1404 emay correspond to storing in a memory pixel coordinates 1402 d, 1404 dof the regions 1402 c, 1404 c and/or an average pixel positioncorresponding to each of the regions 1402 c, 1404 c along with an objectidentifier for the people 1402, 1404.

At step 1524, the camera client 220 a may transmit the first and secondpixel positions (e.g., as pixel positions 1426) to the camera server235. At step 1526, the camera server 225 may apply a homography (e.g.,of homographies 1430) for the camera 305 to the pixel positions todetermine corresponding physical (e.g., global) positions 1428 for thefirst and second people 1402, 1404. Examples of generating and usinghomographies 1430 are described in greater detail above with respect toFIGS. 3A-3T.

Modifications, additions, or omissions may be made to method 1500depicted in FIG. 12. Method 1500 may include more, fewer, or othersteps. For example, steps may be performed in parallel or in anysuitable order. While at times discussed as system 1400, camera client220 a, camera server 235, or components of any of thereof performingsteps, any suitable system or components of the system may perform oneor more steps of the method.

Multi-Sensor Image Tracking on a Local and Global Planes

As described elsewhere in this disclosure (e.g., with respect to FIGS.8-12), tracking people (e.g., or other target objects) in space usingmultiple cameras 305 presents several previously unrecognizedchallenges. This disclosure encompasses not only the recognition ofthese challenges but also unique solutions to these challenges. Forinstance, systems and methods are described in this disclosure thattrack people both locally (e.g., by tracking pixel positions in imagesreceived from each camera 305) and globally (e.g., by tracking physicalpositions on a global plane corresponding to the physical coordinates inthe space). Person tracking may be more reliable when performed bothlocally and globally. For example, if a person is “lost” locally (e.g.,if a camera 305 fails to capture a frame and a person is not detected bythe camera 305), the person may still be tracked globally based on animage from a nearby camera 305 (e.g., the angled-view camera 305 bdescribed with respect to FIG. 11 above), an estimated local position ofthe person determined using a local tracking algorithm, and/or anestimated global position determined using a global tracking algorithm.

As another example, if people appear to merge (e.g., if detectedcontours merge into a single merged contour, as illustrated in view 1416of FIG. 11 above) at one camera 305, an adjacent camera 305 may stillprovide a view in which the people are separate entities (e.g., asillustrated in view 1432 of FIG. 11 above). Thus, information from anadjacent camera 305 may be given priority for person tracking. In someembodiments, if a person tracked via a camera 305 is lost in the localview, estimated pixel positions may be determined using a trackingalgorithm and reported to the camera server 225 for global tracking, atleast until the tracking algorithm determines that the estimatedpositions are below a threshold confidence level.

FIGS. 13A-13C illustrate the use of a tracking subsystem 1600 to track aperson 1602 through the space. FIG. 13A illustrates a portion of thetracking system 132 of FIG. 1 when used to track the position of person1602 based on image data generated by cameras 305 a-c. The position ofperson 1602 is illustrated at three different time points: t₁, t₂, andt₃. Each of the cameras 305 a-c is a camera 305 of FIG. 2, describedabove. Each camera 305 a-c has a corresponding field-of-view 1604 a-c,which corresponds to the portion of the space viewed by the camera 305a-c. As shown in FIG. 13A, each field-of-view 1604 a-c overlaps withthat of the adjacent camera(s) 305 a-c. For example, the adjacentfields-of-view 1604 a-c may overlap by between about 10% and 30%.Cameras 305 a-c generally generate top-view images and transmitcorresponding top-view image feeds 1606 a-c to a tracking subsystem1600.

The tracking subsystem 1600 includes the client(s) 220 and camera server225 of FIG. 2. The tracking system 1600 generally receives top-viewimage feeds 1606 a-c generated by cameras 305 a-c, respectively, anduses the received images (see FIG. 13B) to track a physical (e.g.,global) position of the person 1602 in the space (see FIG. 13C). Eachcamera 305 a-c may be coupled to a corresponding camera client 220 ofthe tracking subsystem 1600. As such, the tracking subsystem 1600 mayinclude local particle filter trackers 1644 for tracking pixel positionsof person 1602 in images generated by cameras 305 a-b, global particlefilter trackers 1646 for tracking physical positions of person 1602 inthe space.

FIG. 13B shows example top-view images 1708 a-c, 1718 a-c, and 1726 a-cgenerated by each of the cameras 305 a-c at times t₁, t₂, and t₃.Certain of the top-view images include representations of the person1602 (i.e., if the person 1602 was in the field-of-view 1604 a-c of thecamera 305 a-c at the time the image 1708 a-c, 1718 a-c, and 1726 a-cwas obtained). For example, at time t₁, images 1708 a-c are generated bycameras 305 a-c, respectively, and provided to the tracking subsystem1600. The tracking subsystem 1600 detects a contour 1710 associated withperson 1602 in image 1708 a. For example, the contour 1710 maycorrespond to a curve outlining the border of a representation of theperson 1602 in image 1708 a (e.g., detected based on color (e.g., RGB)image data at a predefined depth in image 1708 a, as described abovewith respect to FIG. 8). The tracking subsystem 1600 determines pixelcoordinates 1712 a, which are illustrated in this example by thebounding box 1712 b in image 1708 a. Pixel position 1712 c is determinedbased on the coordinates 1712 a. The pixel position 1712 c generallyrefers to the location (i.e., row and column) of the person 1602 in theimage 1708 a. Since the object 1602 is also within the field-of-view1604 b of the second camera 305 b at t₁ (see FIG. 13A), the trackingsystem also detects a contour 1714 in image 1708 b and determinescorresponding pixel coordinates 1716 b (i.e., associated with boundingbox 2416 a) for the object 1602. Pixel position 1716 c is determinedbased on the coordinates 1716 a. The pixel position 1716 c generallyrefers to the pixel location (i.e., row and column) of the person 1602in the image 1708 b. At time t₁, the object 1602 is not in thefield-of-view 1604 c of the third camera 305 c (see FIG. 13A).Accordingly, the tracking subsystem 1600 does not determine pixelcoordinates for the object 1602 based on the image 1708 c received fromthe third camera 305 c.

Turning now to FIG. 13C, the tracking subsystem 1600 (e.g., the cameraserver 225 of the tacking subsystem 1600) may determine a first globalposition 1738 based on the determined pixel positions 1712 c and 1716c(e.g., corresponding to pixel coordinates 1712 a, 1716 a and boundingboxes 1712 b, 1716 b, described above). The first global position 1738corresponds to the position of the person 1602 in the space, asdetermined by the tracking subsystem 1600. In other words, the trackingsubsystem 1600 uses the pixel positions 1712 c, 1716 c determined viathe two cameras 305 a,b to determine a single physical position 1738 forthe person 1602 in the space. For example, a first physical position1712 d may be determined from the pixel position 1712 c associated withbounding box 1712 b using a first homography associating pixelcoordinates in the top-view images generated by the first camera 305 ato physical coordinates in the space. A second physical position 1716 dmay similarly be determined using the pixel position 1716 c associatedwith bounding box 1716 b using a second homography associating pixelcoordinates in the top-view images generated by the second camera 305 bto physical coordinates in the space. In some cases, the trackingsubsystem 1600 may compare the distance between first and secondphysical positions 1712 d and 1716 d to a threshold distance 1648 todetermine whether the positions 1712 d, 1716 d correspond to the sameperson or different people (see, e.g., step 1920 of FIG. 16, describedbelow). The first global position 1738 may be determined as an averageof the first and second physical positions 1710 d, 1714 d. In someembodiments, the global position is determined by clustering the firstand second physical positions 1710 d, 1714 d (e.g., using anyappropriate clustering algorithm). The first global position 1738 maycorrespond to (x,y) coordinates of the position of the person 1602 inthe space.

Returning to FIG. 13A, at time t₂, the object 1602 is withinfields-of-view 1604 a and 1604 b corresponding to cameras 305 a,b. Asshown in FIG. 13B, a contour 1722 is detected in image 1718 b andcorresponding pixel coordinates 1724 a, which are illustrated bybounding box 1724 b, are determined. Pixel position 1724 c is determinedbased on the coordinates 1724 a. The pixel position 1724 c generallyrefers to the location (i.e., row and column) of the person 1602 in theimage 1718 b. However, in this example, the tracking subsystem 1600fails to detect, in image 1718 a from camera 305 a, a contour associatedwith object 1602. This may be because the object 1602 was at the edge ofthe field-of-view 1604 a, because of a lost image frame from feed 1606a, because the position of the person 1602 in the field-of-view 1604 acorresponds to an auto-exclusion zone for camera 305 a (see FIGS. 8-10and corresponding description above), or because of any othermalfunction of camera 305 a and/or the tracking subsystem 1600. In thiscase, the tracking subsystem 1600 may locally (e.g., at the particularclient 220 which is coupled to camera 305 a) estimate pixel coordinates1720 a and/or corresponding pixel position 1720 b for object 1602. Forexample, a local particle filter tracker 1644 for object 1602 in imagesgenerated by camera 305 a may be used to determine the estimated pixelposition 1720 b.

FIGS. 14A,B illustrate the operation of an example particle filtertracker 1644, 1646 (e.g., for determining estimated pixel position 1720a). FIG. 14A illustrates a region 1800 in pixel coordinates or physicalcoordinates of space. For example, region 1800 may correspond to a pixelregion in an image or to a region in physical space. In a first zone1802, an object (e.g., person 1602) is detected at position 1804. Theparticle filter determines several estimated subsequent positions 1806for the object. The estimated subsequent positions 1806 are illustratedas the dots or “particles” in FIG. 14A and are generally determinedbased on a history of previous positions of the object. Similarly,another zone 1808 shows a position 1810 for another object (or the sameobject at a different time) along with estimated subsequent positions1812 of the “particles” for this object.

For the object at position 1804, the estimated subsequent positions 1806are primarily clustered in a similar area above and to the right ofposition 1804, indicating that the particle filter tracker 1644, 1646may provide a relatively good estimate of a subsequent position.Meanwhile, the estimated subsequent positions 1812 are relativelyrandomly distributed around position 1810 for the object, indicatingthat the particle filter tracker 1644, 1646 may provide a relativelypoor estimate of a subsequent position. FIG. 14B shows a distributionplot 1850 of the particles illustrated in FIG. 14A, which may be used toquantify the quality of an estimated position based on a standarddeviation value (σ).

In FIG. 14B, curve 1852 corresponds to the position distribution ofanticipated positions 1806, and curve 1854 corresponds to the positiondistribution of the anticipated positions 1812. Curve 1854 has to arelatively narrow distribution such that the anticipated positions 1806are primarily near the mean position (μ). For example, the narrowdistribution corresponds to the particles primarily having a similarposition, which in this case is above and to right of position 1804. Incontrast, curve 1854 has a broader distribution, where the particles aremore randomly distributed around the mean position (μ). Accordingly, thestandard deviation of curve 1852 (σ₁) is smaller than the standarddeviation curve 1854 (σ₂). Generally, a standard deviation (e.g., eitherσ₁ or σ₂) may be used as a measure of an extent to which an estimatedpixel position generated by the particle filter tracker 1644, 1646 islikely to be correct. If the standard deviation is less than a thresholdstandard deviation (σ_(threshold)), as is the case with curve 1852 andσ₁, the estimated position generated by a particle filter tracker 1644,1646 may be used for object tracking. Otherwise, the estimated positiongenerally is not used for object tracking.

Referring again to FIG. 13C, the tracking subsystem 1600 (e.g., thecamera server 225 of tracking subsystem 1600) may determine a secondglobal position 1740 for the object 1602 in the space based on theestimated pixel position 1720 b associated with estimated bounding box1720 a in frame 1718 a and the pixel position 1724 c associated withbounding box 1724 b from frame 1718 b. For example, a first physicalposition 1720 c may be determined using a first homography associatingpixel coordinates in the top-view images generated by the first camera305 a to physical coordinates in the space. A second physical position1724 d may be determined using a second homography associating pixelcoordinates in the top-view images generated by the second camera 305 bto physical coordinates in the space. The tracking subsystem 1600 (i.e.,camera server 225 of the tracking subsystem 1600) may determine thesecond global position 1740 based on the first and second physicalpositions 1720 c, 1724 d, as described above with respect to time t₁.The second global position 1740 may correspond to (x,y) coordinates ofthe person 1602 in the space.

Turning back to FIG. 13A, at time t₃, the object 1602 is within thefield-of-view 1604 b of camera 305 b and the field-of-view 1604 c ofcamera 305 c. Accordingly, these images 1726 b,c may be used to trackperson 1602. FIG. 13B shows that a contour 1728 and corresponding pixelcoordinates 1730 a, pixel region 1730 b, and pixel position 1730 c aredetermined in frame 1726 b from camera 305 b, while a contour 1732 andcorresponding pixel coordinates 1734 a, pixel region 1734 b, and pixelposition 1734 c are detected in frame 1726 c from camera 305 c. As shownin FIG. 13C and as described in greater detail above for times t₁ andt₂, the tracking subsystem 1600 may determine a third global position1742 for the object 1602 in the space based on the pixel position 1730 cassociated with bounding box 1730 b in frame 1726 b and the pixelposition 1734 c associated with bounding box 1734 b from frame 1726 c.For example, a first physical position 1730 d may be determined using asecond homography associating pixel coordinates in the top-view imagesgenerated by the second camera 305 b to physical coordinates in thespace. A second physical position 1734 d may be determined using a thirdhomography associating pixel coordinates in the top-view imagesgenerated by the third camera 305 c to physical coordinates in thespace. The tracking subsystem 1600 may determine the global position1742 based on the first and second physical positions 1730 d, 1734 d, asdescribed above with respect to times t₁ and t₂.

FIG. 15 is a flow diagram illustrating the tracking of person 1602 inspace the based on top-view images (e.g., images 1708 a-c, 1718 a 0 c,1726 a-c from feeds 1606 a,b, generated by cameras 305 a,b, describedabove. Field-of-view 1604 a of camera 305 a and field-of-view 1604 b ofcameras 305 b generally overlap by a distance 1902. In one embodiment,distance 1902 may be about 10% to 30% of the fields-of-view 1604 a,b. Inthis example, the tracking subsystem 1600 includes the first cameraclient 220 a, the second camera client 220 b, and the camera server 235.Each of the first and second camera clients 220 a,b may be a client 220described above with respect to FIG. 2. The first camera client 220 a iscoupled to the first camera 305 a and configured to track, based on thefirst feed 1606 a, a first pixel position 1312 c of the person 1602. Thesecond camera client 220 b is coupled to the second camera 305 b andconfigured to track, based on the second feed 1606 b, a second pixelposition 1716 c of the same person 1602.

The camera server 225 generally receives pixel positions from clients220 a,b and tracks the global position of the person 1602 in the space.In some embodiments, the camera server 225 employs a global particlefilter tracker 1646 to track a global physical position of the person1602 and one or more other people 1904 in the space). Tracking peopleboth locally (i.e., at the “pixel level” using clients 220 a,b) andglobally (i.e., based on physical positions in the space) improvestracking by reducing and/or eliminating noise and/or other trackingerrors which may result from relying on either local tracking by theclients 220 a,b or global tracking by the camera server 225 alone.

FIG. 15 illustrates a method 1900 implemented by camera clients 220 a,band camera server 235. Camera client 220 a receives the first data feed1606 a from camera 305 a at step 1106 a. The feed may include top-viewimages (e.g., images 1708 a-c, 1718 a-c, 1726 a-c of FIG. 13A). Theimages may be color images, depth images, or color-depth images. In animage from the feed 1606 a (e.g., corresponding to a certain timestamp),the camera client 220 a determines whether a contour is detected at step1108 a. If a contour is detected at the timestamp, the camera client 220a determines a first pixel position 1712 c for the contour at step 1910a. For instance, the first pixel position 1712 c may correspond to pixelcoordinates associated with a bounding box 1712 b determined for thecontour (e.g., using any appropriate object detection algorithm). Asanother example, the camera client 220 a may generate a pixel mask thatoverlays the detected contour and determine pixel coordinates of thepixel mask, as described above with respect to step 1304 of FIG. 10.

If a contour is not detected at step 1108 a, a first particle filtertracker 1644 may be used to estimate a pixel position (e.g., estimatedposition 1720 b), based on a history of previous positions of thecontour 1710, at step 1912 a. For example, the first particle filtertracker 1644 may generate a probability-weighted estimate of asubsequent first pixel position corresponding to the timestamp (e.g., asdescribed above with respect to FIGS. 14A,B). Generally, if theconfidence level (e.g., based on a standard deviation) of the estimatedpixel position 1720 b is below a threshold value (e.g., see FIG. 14B andrelated description above), no pixel position is determined for thetimestamp by the camera client 220 a, and no pixel position is reportedto camera server 225 for the timestamp. This prevents the waste ofprocessing resources which would otherwise be expended by the cameraserver 225 in processing unreliable pixel position data. As describedbelow, the camera server 225 can often still track person 1602, evenwhen no pixel position is provided for a given timestamp, using theglobal particle filter tracker 1646 (see steps 1926, 1932, and 1936below).

The second camera client 220 b receives the second data feed 1606 b fromcamera 305 b at step 1106 b. The same or similar steps to thosedescribed above for camera client 220 a are used to determine a secondpixel position 1716 c for a detected contour 1714 or estimate a pixelposition based on a second particle filter tracker 1644. At step 1108 b,the camera client 220 b determines whether a contour 1714 is detected inan image from feed 1606 b at a given timestamp. If a contour 1714 isdetected at the timestamp, the camera client 220 b determines a firstpixel position 1716 c for the contour 1714 at step 1910 b (e.g., usingany of the approaches described above with respect to step 1910 a). If acontour 1714 is not detected, a second particle filter tracker 1644 maybe used to estimate a pixel position at step 1912 b (e.g., as describedabove with respect to step 1912 a). If the confidence level of theestimated pixel position is below a threshold value (e.g., based on astandard deviation value for the tracker 1644), no pixel position isdetermined for the timestamp by the camera client 220 b, and no pixelposition is reported for the timestamp to the camera server 235.

While steps 1106 a,b-1912 a,b are described as being performed by cameraclient 220 a and 220 b, it should be understood that in someembodiments, a single camera client 220 may receive the first and secondimage feeds 1606 a,b from cameras 305 a,b and perform the stepsdescribed above. Using separate camera clients 220 a,b for separatecameras 305 a,b or sets of cameras 305 may provide redundancy in case ofclient 220 malfunctions (e.g., such that even if one camera client 220fails, feeds from other sensors may be processed by otherstill-functioning clients 220).

At step 1914, the camera server 225 receives the pixel positions 1712 c,1716 c determined by the camera clients 220 a,b. At step 1916, thecamera server 225 may determine a first physical position 1712 d basedon the first pixel position 1712 c determined at step 1910 a orestimated at step 1912 a by the first camera client 220 a. For example,the first physical position 1712 d may be determined using a firsthomography associating pixel coordinates in the top-view imagesgenerated by the first camera 305 a to physical coordinates in thespace. At step 1918, the camera server 225 may determine a secondphysical position 1716 d based on the second pixel position 1716 cdetermined at step 1910 b or estimated at step 1912 b by the firstcamera client 220 b. For instance, the second physical position 1716 dmay be determined using a second homography associating pixelcoordinates in the top-view images generated by the second camera 305 bto physical coordinates in the space.

At step 1920 the camera server 225 determines whether the first andsecond positions 1712 d, 1716 d (from steps 1916 and 1918) are within athreshold distance 1648 (e.g., of about six inches) of each other. Ingeneral, the threshold distance 1648 may be determined based on one ormore characteristics of the system tracking system 132 and/or the person1602 or another target object being tracked. For example, the thresholddistance 1648 may be based on one or more of the distance of the cameras305 a-b from the object, the size of the object, the fields-of-view 1604a-b, the sensitivity of the cameras 305 a-b, and the like. Accordingly,the threshold distance 1648 may range from just over zero inches togreater than six inches depending on these and other characteristics ofthe tracking system 132.

If the positions 1712 d, 1716 d are within the threshold distance 1648of each other at step 1920, the camera server 225 determines that thepositions 1712 d, 1716 d correspond to the same person 1602 at step1922. In other words, the camera server 225 determines that the persondetected by the first camera 305 a is the same person detected by thesecond camera 305 b. This may occur, at a given timestamp, because ofthe overlap 1940 between field-of-view 1604 a and field-of-view 1604 bof cameras 305 a and 108 b, as illustrated in FIG. 15.

At step 1924, the camera server 225 determines a global position 1738(i.e., a physical position in the space) for the object based on thefirst and second physical positions from steps 1916 and 1918. Forinstance, the camera server 225 may calculate an average of the firstand second physical positions 1712 d, 1716 d. In some embodiments, theglobal position 1738 is determined by clustering the first and secondphysical positions 1712 d, 1716 d (e.g., using any appropriateclustering algorithm). At step 1926, a global particle filter tracker1646 is used to track the global (e.g., physical) position 1738 of theperson 1602. An example of a particle filter tracker is described abovewith respect to FIGS. 14A,B. For instance, the global particle filtertracker 1646 may generate probability-weighted estimates of subsequentglobal positions at subsequent times. If a global position 1738 cannotbe determined at a subsequent timestamp (e.g., because pixel positionsare not available from the camera clients 220 a,b), the particle filtertracker 1646 may be used to estimate the position.

If at step 1920 the first and second physical positions 1712 d, 1716 dare not within the threshold distance 1648 from each other, the cameraserver 225 generally determines that the positions correspond todifferent objects 1602, 1104 at step 1928. In other words, the cameraserver 225 may determine that the physical positions determined at steps1916 and 1918 are sufficiently different, or far apart, for them tocorrespond to the first person 1602 and a different second person 1902in the space.

At step 1930, the camera server 225 determines a global position for thefirst object 1602 based on the first physical position 1712 c from step1916. Generally, in the case of having only one physical position 1712 con which to base the global position, the global position is the firstphysical position 1712 c. If other physical positions are associatedwith the first object (e.g., based on data from other cameras 305), theglobal position of the first person 1902 may be an average of thepositions or determined based on the positions using any appropriateclustering algorithm, as described above. At step 1932, a globalparticle filter tracker 1646 may be used to track the first globalposition of the first person 1902, as is also described above.

At step 1934, the camera server 225 determines a global position for thesecond person 1904 based on the second physical position 1716 c fromstep 1918. Generally, in the case of having only one physical position1716 c on which to base the global position, the global position is thesecond physical position 1716 c. If other physical positions areassociated with the second object (e.g., based on data from othercameras 305), the global position of the second person 1904 may be anaverage of the positions or determined based on the positions using anyappropriate clustering algorithm. At step 1936, a global particle filtertracker 1646 is used to track the second global position of the secondobject, as described above.

Modifications, additions, or omissions may be made to the method 1900described above with respect to FIG. 15. The method may include more,fewer, or other steps. For example, steps may be performed in parallelor in any suitable order. While at times discussed as a trackingsubsystem 1600, camera clients 220 a,b, camera server 235, or componentsof any thereof performing steps, any suitable system or components ofthe system may perform one or more steps of the method 1900.

Candidate Lists

When the tracking system 132 is tracking people in the space, it may bechallenging to reliably identify people under certain circumstances suchas when they pass into or near an auto-exclusion zone (see FIGS. 8-10and corresponding description above), when they stand near anotherperson (see FIGS. 11-12 and corresponding description above), and/orwhen one or more of the cameras 305, client(s) 220, and/or camera server225 malfunction. For instance, after a first person becomes close to oreven comes into contact with (e.g., “collides” with) a second person, itmay difficult to determine which person is which (e.g., as describedabove with respect to FIG. 11). Conventional tracking systems may usephysics-based tracking algorithms in an attempt to determine whichperson is which based on estimated trajectories of the people (e.g.,estimated as though the people are marbles colliding and changingtrajectories according to a conservation of momentum, or the like).However, identities of people may be more difficult to track reliably,because movements may be random. As described above, the tracking system132 may employ particle filter tracking for improved tracking of peoplein the space (see e.g., FIGS. 13-16 and the corresponding descriptionabove). However, even with these advancements, the identities of peoplebeing tracked may be difficult to determine at certain times. Thisdisclosure particularly encompasses the recognition that positions ofpeople who are shopping in a store (i.e., moving about a space,selecting items, and picking up the items) are difficult or impossibleto track using previously available technology because movement of thesepeople is random and does not follow a readily defined pattern or model(e.g., such as the physics-based models of previous approaches).Accordingly, there is a lack of tools for reliably and efficientlytracking people (e.g., or other target objects).

This disclosure provides a solution to the problems of previoustechnology, including those described above, by maintaining a record,which is referred to in this disclosure as a “candidate list,” ofpossible person identities, or identifiers (i.e., the usernames, accountnumbers, etc. of the people being tracked), during tracking. A candidatelist is generated and updated during tracking to establish the possibleidentities of each tracked person. Generally, for each possible identityor identifier of a tracked person, the candidate list also includes aprobability that the identity, or identifier, is believed to be correct.The candidate list is updated following interactions (e.g., collisions)between people and in response to other uncertainty events (e.g., a lossof sensor data, imaging errors, intentional trickery, etc.).

In some cases, the candidate list may be used to determine when a personshould be re-identified (e.g., using methods described in greater detailbelow with respect to FIGS. 18-21). Generally, re-identification isappropriate when the candidate list of a tracked person indicates thatthe person's identity is not sufficiently well known (e.g., based on theprobabilities stored in the candidate list being less than a thresholdvalue). In some embodiments, the candidate list is used to determinewhen a person is likely to have exited the space (i.e., with at least athreshold confidence level), and an exit notification is only sent tothe person after there is high confidence level that the person hasexited (see, e.g., view 2030 of FIG. 16, described below). In general,processing resources may be conserved by only performing potentiallycomplex person re-identification tasks when a candidate list indicatesthat a person's identity is no longer known according to pre-establishedcriteria.

FIG. 16 is a flow diagram illustrating how identifiers 2001 a-cassociated with tracked people (e.g., or any other target object) may beupdated during tracking over a period of time from an initial time to t₀a final time t₅ by tracking system 132. People may be tracked usingtracking system 132 based on data from cameras 305, as described above.FIG. 16 depicts a plurality of views 2002, 2016, 2020, 2024, 2028, 2030at different time points during tracking. In some embodiments, views2002, 2016, 2020, 2024, 2028, 2030 correspond to a local frame view(e.g., as described above with respect to FIG. 11) from a camera 305with coordinates in units of pixels (e.g., or any other appropriate unitfor the data type generated by the camera 305). In other embodiments,views 2002, 2016, 2020, 2024, 2028, 2030 correspond to global views ofthe store 100 determined based on data from multiple cameras 305 withcoordinates corresponding to physical positions in the space (e.g., asdetermined using the homographies described in greater detail above withrespect to FIGS. 3A-3T). For clarity and conciseness, the example ofFIG. 16 is described below in terms of global views of the store 100(i.e., a view corresponding to the physical coordinates of the store100).

The tracked object regions 2004, 2008, 2012 correspond to regions of thestore 100 associated with the positions of corresponding people (e.g.,or any other target object) moving through the store 100. For example,each tracked object region 2004, 2008, 2012 may correspond to adifferent person moving about in the store 100. Examples of determiningthe regions 2004, 2008, 2012 are described above, for example, withrespect to FIGS. 10, 11, and 13. As one example, the tracked objectregions 2004, 2008, 2012 may be bounding boxes identified forcorresponding objects in the store 100. As another example, trackedobject regions 2004, 2008, 2012 may correspond to pixel masks determinedfor contours associated with the corresponding objects in the store 100(see, e.g., step 1304 of FIG. 10 for a more detailed description of thedetermination of a pixel mask). Generally, people may be tracked in thestore 100 and regions 2004, 2008, 2012 may be determined using anyappropriate tracking and identification method.

View 2002 at initial time to includes a first tracked object region2004, a second tracked object region 2008, and a third tracked objectregion 2012. The view 2002 may correspond to a representation of thestore 100 from a top view with only the tracked object regions 2004,2008, 2012 shown (i.e., with other objects in the store 100 omitted). Attime to, the identities of all of the people are generally known (e.g.,because the people have recently entered the store 100 and/or becausethe people have not yet been near each other). The first tracked objectregion 2004 is associated with a first candidate list 2006, whichincludes a probability (P_(A)=100%) that the region 2004 (or thecorresponding person being tracked) is associated with a firstidentifier 2001 a. The second tracked object region 2008 is associatedwith a second candidate list 2010, which includes a probability(P_(B)=100%) that the region 2008 (or the corresponding person beingtracked) is associated with a second identifier 2001 b. The thirdtracked object region 2012 is associated with a third candidate list2014, which includes a probability (P_(C)=100%) that the region 2012 (orthe corresponding person being tracked) is associated with a thirdidentifier 2001 c. Accordingly, at time t₁, the candidate lists 2006,2010, 2014 indicate that the identity of each of the tracked objectregions 2004, 2008, 2012 is known with all probabilities having a valueof one hundred percent.

View 2016 shows positions of the tracked objects 2004, 2008, 2012 at afirst time t₁, which is after the initial time t₀. At time t₁, thetracking system detects an event which may cause the identities of thetracked object regions 2004, 2008 to be less certain. In this example,the tracking system 132 detects that the distance 2018 a between thefirst object region 274 and the second object region 2008 is less thanor equal to a threshold distance 2018 b. Because the tracked objectregions were near each other (i.e., within the threshold distance 2018b), there is a non-zero probability that the regions may bemisidentified during subsequent times. The threshold distance 2018 b maybe any appropriate distance, as described above with respect to FIG. 11.For example, the tracking system 132 may determine that the first objectregion 2004 is within the threshold distance 2018 b of the second objectregion 2008 by determining first coordinates of the first object region2004, determining second coordinates of the second object region 2008,calculating a distance 2018 a, and comparing distance 2018 a to thethreshold distance 2018 b. In some embodiments, the first and secondcoordinates correspond to pixel coordinates in an image capturing thefirst and second people, and the distance 2018 a corresponds to a numberof pixels between these pixel coordinates. For example, as illustratedin view 2016 of FIG. 16, the distance 2018 a may correspond to the pixeldistance between centroids of the tracked object regions 2004, 2008. Inother embodiments, the first and second coordinates correspond tophysical, or global, coordinates in the store 100, and the distance 2018a corresponds to a physical distance (e.g., in units of length, such asinches). For example, physical coordinates may be determined using thehomographies described in greater detail above with respect to FIGS.3A-3T.

After detecting that the identities of regions 2004, 2008 are lesscertain (i.e., that the first object region 2004 is within the thresholddistance 2018 b of the second object region 2008), the tracking system132 determines a probability 2017 that the first tracked object region2004 switched identifiers 2001 a-c with the second tracked object region2008. For example, when two contours become close in an image, there isa chance that the identities of the contours may be incorrect duringsubsequent tracking (e.g., because the tracking system 132 may assignthe wrong identifier 2001 a-c to the contours between frames). Theprobability 2017 that the identifiers 2001 a-c switched may bedetermined, for example, by accessing a predefined probability value(e.g., of 50%). In other cases, the probability 2017 may be based on thedistance 2018 a between the object regions 2004, 2008. For example, asthe distance 2018 decreases, the probability 2017 that the identifiers2001 a-c switched may increase. In the example of FIG. 16, thedetermined probability 2017 is 20%, because the object regions 2004,2008 are relatively far apart but there is some overlap between theregions 2004, 2008.

In some embodiments, the tracking system 132 may determine a relativeorientation between the first object region 2004 and the second objectregion 2008, and the probability 2017 that the object regions 2004, 2008switched identifiers 2001 a-c may be based on this relative orientation.The relative orientation may correspond to an angle between a directiona person associated with the first region 2004 is facing and a directiona person associated with the second region 2008 is facing. For example,if the angle between the directions faced by people associated withfirst and second regions 2004, 2008 is near 180° (i.e., such that thepeople are facing in opposite directions), the probability 2017 thatidentifiers 2001 a-c switched may be decreased because this case maycorrespond to one person accidentally backing into the other person.

Based on the determined probability 2017 that the tracked object regions2004, 2008 switched identifiers 2001 a-c (e.g., 20% in this example),the tracking system 132 updates the first candidate list 2006 for thefirst object region 2004. The updated first candidate list 2006 includesa probability (P_(A)=80%) that the first region 2004 is associated withthe first identifier 2001 a and a probability (P_(B)=20%) that the firstregion 2004 is associated with the second identifier 2001 b. The secondcandidate list 2010 for the second object region 2008 is similarlyupdated based on the probability 2017 that the first object region 2004switched identifiers 2001 a-c with the second object region 2008. Theupdated second candidate list 2010 includes a probability (P_(A)=20%)that the second region 2008 is associated with the first identifier 2001a and a probability (P_(B)=80%) that the second region 2008 isassociated with the second identifier 2001 b.

View 2020 shows the object regions 2004, 2008, 2012 at a second timepoint t₂, which follows time t₁. At time t₂, a first personcorresponding to the first tracked region 2004 stands close to a thirdperson corresponding to the third tracked region 2012. In this examplecase, the tracking system 132 detects that the distance 112022 betweenthe first object region 2004 and the third object region 2012 is lessthan or equal to the threshold distance 2018 b (i.e., the same thresholddistance 2018 b described above with respect to view 2016). Afterdetecting that the first object region 2004 is within the thresholddistance 2018 b of the third object region 2012, the tracking system 132determines a probability 2021 that the first tracked object region 2004switched identifiers 2001 a-c with the third tracked object region 2012.As described above, the probability 2021 that the identifiers 2001 a-cswitched may be determined, for example, by accessing a predefinedprobability value (e.g., of 50%). In some cases, the probability 2021may be based on the distance 112022 between the object regions 2004,2012. For example, since the distance 112022 is greater than distance2018 a (from view 2016, described above), the probability 2021 that theidentifiers 2001 a-c switched may be greater at time t₁ than at time t₂.In the example of view 2020 of FIG. 16, the determined probability 2021is 10% (which is smaller than the switching probability 2717 of 20%determined at time t₁).

Based on the determined probability 2021 that the tracked object regions2004, 2012 switched identifiers 2001 a-c (e.g., of 10% in this example),the tracking system 132 updates the first candidate list 2006 for thefirst object region 2004. The updated first candidate list 2006 includesa probability (P_(A)=73%) that the first object region 2004 isassociated with the first identifier 2001 a, a probability (P_(B)=17%)that the first object region 2004 is associated with the secondidentifier 2001 b, and a probability (P_(C)=10%) that the first objectregion 2004 is associated with the third identifier 2001 c. The thirdcandidate list 2014 for the third object region 2012 is similarlyupdated based on the probability 2021 that the first object region 2004switched identifiers 2001 a-c with the third object region 2012. Theupdated third candidate list 2014 includes a probability (P_(A)=7%) thatthe third object region 2012 is associated with the first identifier2001 a, a probability (P_(B)=3%) that the third object region 2012 isassociated with the second identifier 2001 b, and a probability(P_(C)=90%) that the third object region 2012 is associated with thethird identifier 2001 c. Accordingly, even though the third objectregion 2012 never interacted with (e.g., came within the thresholddistance 2018 b of) the second object region 2008, there is still anon-zero probability (P_(B)=3%) that the third object region 2012 isassociated with the second identifier 2001 b, which was originallyassigned (at time to) to the second object region 2008. In other words,the uncertainty in object identity that was detected at time t₁ ispropagated to the third object region 2012 via the interaction withregion 2004 at time t₂. This unique “propagation effect” facilitatesimproved object identification and can be used to narrow the searchspace (e.g., the number of possible identifiers 2001 a-c that may beassociated with a tracked object region 2004, 2008, 2012) when objectre-identification is needed (as described in greater detail below andwith respect to FIGS. 18-21).

View 2024 shows third object region 2012 and an unidentified objectregion 2026 at a third time point t₃, which follows time t₂. At time t₃,the first and second people associated with regions 2004, 2008 come intocontact (e.g., or “collide”) or are otherwise so close to one anotherthat the tracking system 132 cannot distinguish between the people. Forexample, contours detected for determining the first object region 2004and the second object region 2008 may have merged resulting in thesingle unidentified object region 2026. Accordingly, the position ofobject region 2026 may correspond to the position of one or both ofobject regions 2004 and 2008. At time t₃, the tracking system 132 maydetermine that the first and second object regions 2004, 2008 are nolonger detected because a first contour associated with the first objectregion 2004 is merged with a second contour associated with the secondobject region 2008.

The tracking system 132 may wait until a subsequent time t₄ (shown inview 2028) when the first and second object regions 2004, 2008 are againdetected before the candidate lists 2006, 2010 are updated. Time t₄generally corresponds to a time when the first and second peopleassociated with regions 2004, 2008 have separated from each other suchthat each person can be tracked in the store 100. Following a mergingevent such as is illustrated in view 2024, the probability 2025 thatregions 2004 and 2008 have switched identifiers 2001 a-c may be 50%. Attime t₄, updated candidate list 2006 includes an updated probability(P_(A)=60%) that the first object region 2004 is associated with thefirst identifier 2001 a, an updated probability (P_(B)=35%) that thefirst object region 2004 is associated with the second identifier 2001b, and an updated probability (P_(C)=5%) that the first object region2004 is associated with the third identifier 2001 c. Updated candidatelist 2010 includes an updated probability (P_(A)=33%) that the secondobject region 2008 is associated with the first identifier 2001 a, anupdated probability (P_(B)=62%) that the second object region 2008 isassociated with the second identifier 2001 b, and an updated probability(P_(C)=5%) that the second object region 2008 is associated with thethird identifier 2001 c. Candidate list 2014 is unchanged.

Still referring to view 2028, the tracking system 132 may determine thata highest value probability of a candidate list is less than a thresholdvalue (e.g., P_(threshold)=70%). In response to determining that thehighest probability of the first candidate list 2006 is less than thethreshold value, the corresponding object region 2004 may bere-identified (e.g., using any method of re-identification described inthis disclosure, for example, with respect to FIGS. 18-21). Forinstance, the first object region 2004 may be re-identified because thehighest probability (P_(A)=60%) is less than the threshold probability(P_(threshold)=70%). The tracking system 132 may extract features, ordescriptors, associated with observable characteristics of the firstperson (or corresponding contour) associated with the first objectregion 2004. The observable characteristics may be a height of theobject (e.g., determined from depth data received from a sensor), acolor associated with an area inside the contour (e.g., based on colorimage data from a camera 305), a width of the object, an aspect ratio(e.g., width/length) of the object, a volume of the object (e.g., basedon depth data from camera 305), or the like. Examples of otherdescriptors are described in greater detail below with respect to FIG.19. As described in greater detail below, a texture feature (e.g.,determined using a local binary pattern histogram (LBPH) algorithm) maybe calculated for the person. Alternatively or additionally, anartificial neural network may be used to associate the person with thecorrect identifier 2001 a-c (e.g., as described in greater detail belowwith respect to FIG. 18-21).

Using the candidate lists 2006, 2010, 2014 may facilitate more efficientre-identification than was previously possible because, rather thanchecking all possible identifiers 2001 a-c (e.g., and other identifiersof people in store 100 not illustrated in FIG. 16) for a region 2004,2008, 2012 that has an uncertain identity, the tracking system 132 mayidentify a subset of all the other identifiers 2001 a-c that are mostlikely to be associated with the unknown region 2004, 2008, 2012 andonly compare descriptors of the unknown region 2004, 2008, 2012 todescriptors associated with the subset of identifiers 2001 a-c. In otherwords, if the identity of a tracked person is not certain, the trackingsystem 132 may only check to see if the person is one of the few peopleindicated in the person's candidate list, rather than comparing theunknown person to all of the people in the store 100. For example, onlyidentifiers 2001 a-c associated with a non-zero probability, or aprobability greater than a threshold value, in the candidate list 2006are likely to be associated with the correct identifier 2001 a-c of thefirst region 2004. In some embodiments, the subset may includeidentifiers 2001 a-c from the first candidate list 2006 withprobabilities that are greater than a threshold probability value (e.g.,of 10%). Thus, the tracking system 132 may compare descriptors of theperson associated with region 2004 to predetermined descriptorsassociated with the subset. As described in greater detail below withrespect to FIGS. 18-29, the predetermined features (or descriptors) maybe determined when a person enters the store 100 and associated with theknown identifier 2001 a-c of the person during the entrance time period(i.e., before any events may cause the identity of the person to beuncertain. In the example of FIG. 16, the object region 2008 may also bere-identified at or after time t₄ because the highest probabilityP_(B)=62% is less than the example threshold probability of 70%.

View 2030 corresponds to a time t₅ at which only the person associatedwith object region 2012 remains within the store 100. View 2030illustrates how the candidate lists 2006, 2010, 2014 can be used toensure that people only receive an exit notification 2034 when thetracking system 132 is certain the person has exited the store 100. Inthese embodiments, the tracking system 132 may be configured to transmitan exit notification 2034 to devices associated with these people whenthe probability that a person has exited the store 100 is greater thanan exit threshold (e.g., P_(exit)=95% or greater).

An exit notification 2034 is generally sent to the device of a personand includes an acknowledgement that the tracking system 132 hasdetermined that the person has exited the store 100. For example, if thestore 100 is a store, the exit notification 2034 provides a confirmationto the person that the tracking system 132 knows the person has exitedthe store and is, thus, no longer shopping. This may provide assuranceto the person that the tracking system 132 is operating properly and isno longer assigning items to the person or incorrectly charging theperson for items that he/she did not intend to purchase.

As people exit the store 100, the tracking system 132 may maintain arecord 2032 of exit probabilities to determine when an exit notification2034 should be sent. In the example of FIG. 16, at time t₅ (shown inview 2030), the record 2032 includes an exit probability(P_(A,exit)=93%) that a first person associated with the first objectregion 2004 has exited the store 100. Since P_(A,exit) is less than theexample threshold exit probability of 95%, an exit notification 2034would not be sent to the first person (e.g., to his/her device). Thus,even though the first object region 2004 is no longer detected in thestore 100, an exit notification 2034 is not sent, because there is stilla chance that the first person is still in the store 100 (i.e., becauseof identity uncertainties that are captured and recorded via thecandidate lists 2006, 2010, 2014). This prevents a person from receivingan exit notification 2034 before he/she has exited the store 100. Therecord 2032 includes an exit probability (P_(B,exit)=97%) that thesecond person associated with the second object region 2008 has exitedthe store 100. Since P_(B,exit) is greater than the threshold exitprobability of 95%, an exit notification 2034 is sent to the secondperson (e.g., to his/her device). The record 2032 also includes an exitprobability (P_(C,exit)=10%) that the third person associated with thethird object region 2012 has exited the store 100. Since P_(C,exit) isless than the threshold exit probability of 95%, an exit notification2034 is not sent to the third person (e.g., to his/her device).

FIG. 17 is a flowchart of a method 2100 for creating and/or maintainingcandidate lists 2006, 2010, 2014 by tracking system 132. Method 2100generally facilitates improved identification of tracked people (e.g.,or other target objects) by maintaining candidate lists 2006, 2010, 2014which, for a given tracked person, or corresponding tracked objectregion (e.g., region 2004, 2008, 2012), include possible identifiers2001 a-c for the object and a corresponding probability that eachidentifier 2001 a-c is correct for the person. By maintaining candidatelists 2006, 2010, 2014 for tracked people, the people may be moreeffectively and efficiently identified during tracking. For example,costly person re-identification (e.g., in terms of system resourcesexpended) may only be used when a candidate list indicates that aperson's identity is sufficiently uncertain.

Method 2100 may begin at step 2102 where image frames are received fromone or more cameras 305. At step 2104, the tracking system 132 uses thereceived frames to track objects in the store 100. In some embodiments,tracking is performed using one or more of the unique tools described inthis disclosure (e.g., with respect to FIGS. 13-15). However, ingeneral, any appropriate method of sensor-based object tracking may beemployed.

At step 2106, the tracking system 132 determines whether a first personis within a threshold distance 2018 b of a second person. This case maycorrespond to the conditions shown in view 2016 of FIG. 16, describedabove, where first object region 2004 is distance 2018 a away fromsecond object region 2008. As described above, the distance 2018 a maycorrespond to a pixel distance measured in a frame or a physicaldistance in the store 100 (e.g., determined using a homographyassociating pixel coordinates to physical coordinates in the store 100).If the first and second people are not within the threshold distance2018 b of each other, the system 132 continues tracking objects in thestore 100 (i.e., by returning to step 2104).

However, if the first and second people are within the thresholddistance 2018 b of each other, method 2100 proceeds to step 2108, wherethe probability 2017 that the first and second people switchedidentifiers 2001 a-c is determined. As described above, the probability2017 that the identifiers 2001 a-c switched may be determined, forexample, by accessing a predefined probability value (e.g., of 50%). Insome embodiments, the probability 2017 is based on the distance 2018 abetween the people (or corresponding object regions 2004, 2008), asdescribed above. In some embodiments, as described above, the trackingsystem 132 determines a relative orientation between the first personand the second person, and the probability 2017 that the people (orcorresponding object regions 2004, 2008) switched identifiers 2001 a-cis determined, at least in part, based on this relative orientation.

At step 2110, the candidate lists 2006, 2010 for the first and secondpeople (or corresponding object regions 2004, 2008) are updated based onthe probability 2017 determined at step 2108. For instance, as describedabove, the updated first candidate list 2006 may include a probabilitythat the first object is associated with the first identifier 2001 a anda probability that the first object is associated with the secondidentifier 2001 b. The second candidate list 2010 for the second personis similarly updated based on the probability 2017 that the first objectswitched identifiers 2001 a-c with the second object (determined at step2108). The updated second candidate list 2010 may include a probabilitythat the second person is associated with the first identifier 2001 aand a probability that the second person is associated with the secondidentifier 2001 b.

At step 2112, the tracking system 132 determines whether the firstperson (or corresponding region 2004) is within a threshold distance2018 b of a third object (or corresponding region 2012). This case maycorrespond, for example, to the conditions shown in view 2020 of FIG.16, described above, where first object region 2004 is distance 112022away from third object region 2012. As described above, the thresholddistance 2018 b may correspond to a pixel distance measured in a frameor a physical distance in the store 100 (e.g., determined using anappropriate homography associating pixel coordinates to physicalcoordinates in the store 100).

If the first and third people (or corresponding regions 2004 and 2012)are within the threshold distance 2018 b of each other, method 2100proceeds to step 2114, where the probability 2021 that the first andthird people (or corresponding regions 2004 and 2012) switchedidentifiers 2001 a-c is determined. As described above, this probability2021 that the identifiers 2001 a-c switched may be determined, forexample, by accessing a predefined probability value (e.g., of 50%). Theprobability 2021 may also or alternatively be based on the distance112022 between the objects 2727 and/or a relative orientation of thefirst and third people, as described above. At step 2116, the candidatelists 2006, 2010 for the first and third people (or correspondingregions 2004, 2012) are updated based on the probability 2021 determinedat step 2108. For instance, as described above, the updated firstcandidate list 2006 may include a probability that the first person isassociated with the first identifier 2001 a, a probability that thefirst person is associated with the second identifier 2001 b, and aprobability that the first object is associated with the thirdidentifier 2001 c. The third candidate list 2014 for the third person issimilarly updated based on the probability 2021 that the first personswitched identifiers with the third person (i.e., determined at step2114). The updated third candidate list 2014 may include, for example, aprobability that the third object is associated with the firstidentifier 2001 a, a probability that the third object is associatedwith the second identifier 2001 b, and a probability that the thirdobject is associated with the third identifier 2001 c. Accordingly, ifthe steps of method 2100 proceed in the example order illustrated inFIG. 17, the candidate list 2014 of the third person includes a non-zeroprobability that the third object is associated with the secondidentifier 2001 b, which was originally associated with the secondperson.

If, at step 2112, the first and third people (or corresponding regions2004 and 2012) are not within the threshold distance 2018 b of eachother, the tracking system 132 generally continues tracking people inthe store 100. For example, the tracking system 132 may proceed to step2118 to determine whether the first person is within a thresholddistance of an n^(th) person (i.e., some other person in the store 100).At step 2120, the tracking system 132 determines the probability thatthe first and n^(th) people switched identifiers 2001 a-c, as describedabove, for example, with respect to steps 2108 and 2114. At step 2122,the candidate lists for the first and n^(th) people are updated based onthe probability determined at step 2120, as described above, forexample, with respect to steps 2110 and 2116 before method 2100 ends.If, at step 2118, the first person is not within the threshold distanceof the n^(th) person, the method 2100 proceeds to step 2124.

At step 2124, the tracking system 132 determines if a person has exitedthe store 100. For instance, as described above, the tracking system 132may determine that a contour associated with a tracked person is nolonger detected for at least a threshold time period (e.g., of about 30seconds or more). The system 132 may additionally determine that aperson exited the store 100 when a person is no longer detected and alast determined position of the person was at or near an exit position(e.g., near a door leading to a known exit from the store 100). If aperson has not exited the store 100, the tracking system 132 continuesto track people (e.g., by returning to step 2102).

If a person has exited the store 100, the tracking system 132 calculatesor updates record 2032 of probabilities that the tracked objects haveexited the store 100 at step 2126. As described above, each exitprobability of record 2032 generally corresponds to a probability that aperson associated with each identifier 2001 a-c has exited the store100. At step 2128, the tracking system 132 determines if a combined exitprobability in the record 2032 is greater than a threshold value (e.g.,of 95% or greater). If a combined exit probability is not greater thanthe threshold, the tracking system 132 continues to track objects (e.g.,by continuing to step 2118).

If an exit probability from record 2032 is greater than the threshold, acorresponding exit notification 2034 may be sent to the person linked tothe identifier 200 la-c associated with the probability at step 2130, asdescribed above with respect to view 2030 of FIG. 16. This may preventor reduce instances where an exit notification 2034 is sent prematurelywhile an object is still in the store 100. For example, it may bebeneficial to delay sending an exit notification 2034 until there is ahigh certainty that the associated person is no longer in the store 100.In some cases, several tracked people must exit the store 100 before anexit probability in record 2032 for a given identifier 2001 a-c issufficiently large for an exit notification 2034 to be sent to theperson (e.g., to a device associated with the person).

Modifications, additions, or omissions may be made to method 2100depicted in FIG. 17. Method 2100 may include more, fewer, or othersteps. For example, steps may be performed in parallel or in anysuitable order. While at times discussed as tracking system 132 orcomponents thereof performing steps, any suitable system or componentsof the system may perform one or more steps of the method 2100.

Person Re-Identification

As described above, in some cases, the identity of a tracked person canbecome unknown (e.g., when the people become closely spaced or“collide”, or when the candidate list of a person indicates the person'sidentity is not known, as described above with respect to FIGS. 16-17),and the person may need to be re-identified. This disclosurecontemplates a unique approach to efficiently and reliablyre-identifying people by the tracking system 132. For example, ratherthan relying entirely on resource-expensive machine learning-basedapproaches to re-identify people, a more efficient and speciallystructured approach may be used where “lower-cost” descriptors relatedto observable characteristics (e.g., height, color, width, volume, etc.)of people are used first for person re-identification. “Higher-cost”descriptors (e.g., determined using artificial neural network models)are only used when the lower-cost methods cannot provide reliableresults. For instance, in some embodiments, a person may first bere-identified based on his/her height, hair color, and/or shoe color.However, if these descriptors are not sufficient for reliablyre-identifying the person (e.g., because other people being tracked havesimilar characteristics), progressively higher-level approaches may beused (e.g., involving artificial neural networks that are trained torecognize people) which may be more effective at person identificationbut which generally involve the use of more processing resources.

As an example, each person's height may be used initially forre-identification. However, if another person in the store 100 has asimilar height, a height descriptor may not be sufficient forre-identifying the people (e.g., because it is not possible todistinguish between people with a similar heights based on heightalone), and a higher-level approach may be used (e.g., using a textureoperator or an artificial neural network to characterize the person). Insome embodiments, if the other person with a similar height has neverinteracted with the person being re-identified (e.g., as recorded ineach person's candidate list—see FIG. 16 and corresponding descriptionabove), height may still be an appropriate feature for re-identifyingthe person (e.g., because the other person with a similar height is notassociated with a candidate identity of the person being re-identified).

FIG. 18 illustrates a tracking subsystem 2200 configured to track people(e.g., and/or other target objects) based on sensor data 2204 receivedfrom one or more cameras 305. In general, the tracking subsystem 2200may include one or both of the camera server 225 and the cameraclient(s) 220 of FIG. 2, described above. Tracking subsystem 2200 may beimplemented as described with respect to FIG. 7. Tracking subsystem 2200may track object positions 11202, over a period of time using sensordata 2204 (e.g., top-view images) generated by at least one of cameras305. Object positions 11202 may correspond to local pixel positions(e.g., pixel positions 1126,1134 of FIG. 11) determined at a singlecamera 305 and/or global positions corresponding to physical positions(e.g., positions 1128 of FIG. 11) in the store 100 (e.g., using thehomographies described above with respect to FIGS. 3A-3T). In somecases, object positions 11202 may correspond to regions detected in animage, or in the store 100, that are associated with the location of acorresponding person (e.g., regions 2004, 2008, 2012 of FIG. 16,described above). People may be tracked and corresponding positions11202 may be determined, for example, based on pixel coordinates ofcontours detected in top-view images generated by camera(s) 305.Examples of contour-based detection and tracking are described above,for example, with respect to FIGS. 13 and 16. However, in general, anyappropriate method of sensor-based tracking may be used to determinepositions 11202.

For each object position 11202, the subsystem 2200 maintains acorresponding candidate list 2206 (e.g., as described above with respectto FIG. 16). The candidate lists 2206 are generally used to maintain arecord of the most likely identities of each person being tracked (i.e.,associated with positions 11202). Each candidate list 2206 includesprobabilities which are associated with identifiers 2208 of people thathave entered the store 100. The identifiers 2208 may be any appropriaterepresentation (e.g., an alphanumeric string, or the like) foridentifying a person (e.g., a username, name, account number, or thelike associated with the person being tracked). In some embodiments, theidentifiers 2208 may be anonymized (e.g., using hashing or any otherappropriate anonymization technique).

Each of the identifiers 2208 is associated with one or morepredetermined descriptors 2210. The predetermined descriptors 2210generally correspond to information about the tracked people that can beused to re-identify the people when necessary (e.g., based on thecandidate lists 2206). The predetermined descriptors 2210 may includevalues associated with observable and/or calculated characteristics ofthe people associated with the identifiers 2208. For instance, thedescriptors 2210 may include heights, hair colors, clothing colors, andthe like. As described in greater detail below, the predetermineddescriptors 2210 are generally determined by the tracking subsystem 2200during an initial time period (e.g., when a person associated with agiven tracked position 11202 enters the space) and are used tore-identify people associated with tracked positions 11202 whennecessary (e.g., based on candidate lists 2206).

When re-identification is needed (or periodically during tracking) for agiven person at position 11202, the tracking subsystem 2200 maydetermine measured descriptors 2212 for the person associated with theposition 11202. FIG. 19 illustrates the determination of descriptors2210, 2212 based on a top-view depth image 2302 received from a camera305. A representation 2204 a of a person corresponding to the trackedobject position 11202 is observable in the image 2302. The trackingsubsystem 2200 may detect a contour 2304 b associated with therepresentation 2304 a. The contour 2304 b may correspond to a boundaryof the representation 2304 a (e.g., determined at a given depth in image2302). Tracking subsystem 2200 generally determines descriptors 2210,2212 based on the representation 2304 a and/or the contour 2304 b. Insome cases, the representation 2304 b appears within a predefinedregion-of-interest 2306 of the image 2302 in order for descriptors 2210,2212 to be determined by the tracking subsystem 2200. This mayfacilitate more reliable descriptor 2210, 2212 determination, forexample, because descriptors 2210, 2212 may be more reproducible and/orreliable when the person being imaged is located in the portion of thesensor's field-of-view that corresponds to this region-of-interest 2306.For example, descriptors 2210, 2212 may have more consistent values whenthe person is imaged within the region-of-interest 2306.

Descriptors 2210, 2212 determined in this manner may include, forexample, observable descriptors 2308 and calculated descriptors 2310.For example, the observable descriptors 2308 may correspond tocharacteristics of the representation 2304 a and/or contour 2304 b whichcan be extracted from the image 2302 and which correspond to observablefeatures of the person. Examples of observable descriptors 2308 includea height descriptor 2312 (e.g., a measure of the height in pixels orunits of length) of the person based on representation 2304 a and/orcontour 2304 b), a shape descriptor 2314 (e.g., width, length, aspectratio, etc.) of the representation 2304 a and/or contour 2304 b, avolume descriptor 2316 of the representation 2304 a and/or contour 2304b, a color descriptor 2318 of representation 2304 a (e.g., a color ofthe person's hair, clothing, shoes, etc.), an attribute descriptor 2320associated with the appearance of the representation 2304 a and/orcontour 2304 b (e.g., an attribute such as “wearing a hat,” “carrying achild,” “pushing a stroller or cart,”), and the like.

In contrast to the observable descriptors 2308, the calculateddescriptors 2310 generally include values (e.g., scalar or vectorvalues) which are calculated using the representation 2304 a and/orcontour 2304 b and which do not necessarily correspond to an observablecharacteristic of the person. For example, the calculated descriptors2310 may include image-based descriptors 2322 and model-baseddescriptors 2324. Image-based descriptors 2322 may, for example, includeany descriptor values (i.e., scalar and/or vector values) calculatedfrom image 2302. For example, a texture operator such as a local binarypattern histogram (LBPH) algorithm may be used to calculate a vectorassociated with the representation 2304 a. This vector may be stored asa predetermined descriptor 2210 and measured at subsequent times as adescriptor 2212 for re-identification. Since the output of a textureoperator, such as the LBPH algorithm may be large (i.e., in terms of theamount of memory required to store the output), it may be beneficial toselect a subset of the output that is most useful for distinguishingpeople. Accordingly, in some cases, the tracking subsystem 2200 mayselect a portion of the initial data vector to include in the descriptor2210, 2212. For example, principal component analysis may be used toselect and retain a portion of the initial data vector that is mostuseful for effective person re-identification.

In contrast to the image-based descriptors 2322, model-based descriptors2324 are generally determined using a predefined model, such as anartificial neural network. For example, a model-based descriptor 2324may be the output (e.g., a scalar value or vector) output by anartificial neural network trained to recognize people based on theircorresponding representation 2304 a and/or contour 2304 b in top-viewimage 2302. For example, a Siamese neural network may be trained toassociate representations 2304 a and/or contours 2304 b in top-viewimages 2302 with corresponding identifiers 2208 and subsequentlyemployed for re-identification 2929.

Returning to FIG. 18, the descriptor comparator 2214 of the trackingsubsystem 2200 may be used to compare the measured descriptor 2212 tocorresponding predetermined descriptors 2210 in order to determine thecorrect identity of a person being tracked. For example, the measureddescriptor 2212 may be compared to a corresponding predetermineddescriptor 2210 in order to determine the correct identifier 2208 forthe person at position 11202. For instance, if the measured descriptor2212 is a height descriptor 2312, it may be compared to predeterminedheight descriptors 2210 for identifiers 2208, or a subset of theidentifiers 2208 determined using the candidate list 2206. Comparing thedescriptors 2210, 2212 may involve calculating a difference betweenscalar descriptor values (e.g., a difference in heights 2312, volumes2318, etc.), determining whether a value of a measured descriptor 2212is within a threshold range of the corresponding predetermineddescriptor 2210 (e.g., determining if a color value 2318 of the measureddescriptor 2212 is within a threshold range of the color value 2318 ofthe predetermined descriptor 2210), determining a cosine similarityvalue between vectors of the measured descriptor 2212 and thecorresponding predetermined descriptor 2210 (e.g., determining a cosinesimilarity value between a measured vector calculated using a textureoperator or neural network and a predetermined vector calculated in thesame manner). In some embodiments, only a subset of the predetermineddescriptors 2210 are compared to the measured descriptor 2212. Thesubset may be selected using the candidate list 2206 for the person atposition 11202 that is being re-identified. For example, the person'scandidate list 2206 may indicate that only a subset (e.g., two, three,or so) of a larger number of identifiers 2208 are likely to beassociated with the tracked object position 11202 that requiresre-identification.

When the correct identifier 2208 is determined by the descriptorcomparator 2214, the comparator 2214 may update the candidate list 2206for the person being re-identified at position 11202 (e.g., by sendingupdate 2216). In some cases, a descriptor 2212 may be measured for anobject that does not require re-identification (e.g., a person for whichthe candidate list 2206 indicates there is 100% probability that theperson corresponds to a single identifier 2208). In these cases,measured identifiers 2212 may be used to update and/or maintain thepredetermined descriptors 2210 for the person's known identifier 2208(e.g., by sending update 2218). For instance, a predetermined descriptor2210 may need to be updated if a person associated with the position11202 has a change of appearance while moving through the store 100(e.g., by adding or removing an article of clothing, by assuming adifferent posture, etc.).

FIG. 20A illustrates positions over a period of time of tracked people2402, 2404, 2406, during an example operation of tracking system 2200.The first person 2402 has a corresponding trajectory 2408 represented bythe solid line in FIG. 20A. Trajectory 2408 corresponds to the historyof positions of person 2402 in the store 100 during the period of time.Similarly, the second person 2404 has a corresponding trajectory 2410represented by the dashed-dotted line in FIG. 20A. Trajectory 2410corresponds to the history of positions of person 2404 in the store 100during the period of time. The third person 2406 has a correspondingtrajectory 2412 represented by the dotted line in FIG. 20A. Trajectory2412 corresponds to the history of positions of person 2406 in the store100 during the period of time.

When each of the people 2402, 2404, 2406 first enter the store 100(e.g., when they are within region 2414), predetermined descriptors 2210are generally determined for the people 2402, 2404, 2406 and associatedwith the identifiers 2208 of the people 2402, 2404, 2406. Thepredetermined descriptors 2210 are generally accessed when the identityof one or more of the people 2402, 2404, 2406 is not sufficientlycertain (e.g., based on the corresponding candidate list 2206 and/or inresponse to a “collision event,” as described below) in order tore-identify the person 2402, 2404, 2406. For example, re-identificationmay be needed following a “collision event” between two or more of thepeople 2402, 2404, 2406. A collision event typically corresponds to animage frame in which contours associated with different people merge toform a single contour (e.g., the detection of merged contour 1120 shownin FIG. 11 may correspond to detecting a collision event). In someembodiments, a collision event corresponds to a person being locatedwithin a threshold distance of another person (see, e.g., distance 2018a and 112022 in FIG. 16 and the corresponding description above). Moregenerally, a collision event may correspond to any event that results ina person's candidate list 2206 indicating that re-identification isneeded (e.g., based on probabilities stored in the candidate list2206—see FIGS. 16-17 and the corresponding description above).

In the example of FIG. 20A, when the people 2402, 2404, 2406 are withinregion 2414, the tracking subsystem 2200 may determine a first heightdescriptor 2312 associated with a first height of the first person 2402,a first contour descriptor 2314 associated with a shape of the firstperson 2402, a first anchor descriptor 2324 corresponding to a firstvector generated by an artificial neural network for the first person2402, and/or any other descriptors 2210 described with respect to FIG.19 above. Each of these descriptors is stored for use as a predetermineddescriptor 2210 for re-identifying the first person 2402. Thesepredetermined descriptors 2210 are associated with the first identifier(i.e., of identifiers 2208) of the first person 2402. When the identityof the first person 2402 is certain (e.g., prior to the first collisionevent at position 2416), each of the descriptors 2210 described abovemay be determined again to update the predetermined descriptors 2210.For example, if person 2402 moves to a position in the store 100 thatallows the person 2402 to be within a desired region-of-interest (e.g.,region-of-interest 2306 of FIG. 19), new descriptors 2212 may bedetermined. The tracking subsystem 2200 may use these new descriptors2212 to update the previously determined descriptors 2210 (e.g., seeupdate 2218 of FIG. 18). By intermittently updating the predetermineddescriptors 2210, changes in the appearance of people being tracked canbe accounted for (e.g., if a person puts on or removes an article ofclothing, assumes a different posture, etc.).

At a first timestamp associated with a time t₁, the tracking subsystem2200 detects a collision event between the first person 2402 and thirdperson 2406 at position 2416 illustrated in FIG. 20A. For example, thecollision event may correspond to a first tracked position of the firstperson 2402 being within a threshold distance of a second trackedposition of the third person 2406 at the first timestamp. In someembodiments, the collision event corresponds to a first contourassociated with the first person 2402 merging with a third contourassociated with the third person 2406 at the first timestamp. Moregenerally, the collision event may be associated with any occurrencewhich causes a highest value probability of a candidate list associatedwith the first person 2402 and/or the third person 2406 to fall below athreshold value (e.g., as described above with respect to view 2028 ofFIG. 16). In other words, any event causing the identity of person 2402to become uncertain may be considered a collision event.

After the collision event is detected, the tracking subsystem 2200receives a top-view image (e.g., top-view image 2302 of FIG. 19) fromcamera 305. The tracking subsystem 2200 determines, based on thetop-view image, a first descriptor for the first person 2402. Asdescribed above, the first descriptor includes at least one valueassociated with an observable, or calculated, characteristic of thefirst person 2404 (e.g., of representation 2304 a and/or contour 2304 bof FIG. 19). In some embodiments, the first descriptor may be a“lower-cost” descriptor that requires relative few processing resourcesto determine, as described above. For example, the tracking subsystem2200 may be able to determine a lower-cost descriptor more efficientlythan it can determine a higher-cost descriptor (e.g., a model-baseddescriptor 2324 described above with respect to FIG. 19). For instance,a first number of processing cores used to determine the firstdescriptor may be less than a second number of processing cores used todetermine a model-based descriptor 2324 (e.g., using an artificialneural network). Thus, it may be beneficial to re-identify a person,whenever possible, using a lower-cost descriptor whenever possible.

However, in some cases, the first descriptor may not be sufficient forre-identifying the first person 2402. For example, if the first person2402 and the third person 2406 correspond to people with similarheights, a height descriptor 2312 generally cannot be used todistinguish between the people 2402, 2406. Accordingly, before the firstdescriptor 2212 is used to re-identify the first person 2402, thetracking subsystem 2200 may determine whether certain criteria aresatisfied for distinguishing the first person 2402 from the third person2406 based on the first descriptor 2212. In some embodiments, thecriteria are not satisfied when a difference, determined during a timeinterval associated with the collision event (e.g., at a time at or neartime t₁), between the descriptor 2212 of the first person 2402 and acorresponding descriptor 2212 of the third person 2406 is less than aminimum value.

FIG. 20B illustrates the evaluation of these criteria based on thehistory of descriptor values for people 2402 and 2406 over time. Plot2450, shown in FIG. 20B, shows a first descriptor value 2452 for thefirst person 2402 over time and a second descriptor value 2454 for thethird person 2406 over time. In general, descriptor values may fluctuateover time because of changes in the environment, the orientation ofpeople relative to cameras 305, sensor variability, changes inappearance, etc. The descriptor values 2452, 2454 may be associated witha shape descriptor 2314, a volume 2316, a contour-based descriptor 2322,or the like, as described above with respect to FIG. 19. At time t₁, thedescriptor values 2452, 2454 have a relatively large difference 2456that is greater than the threshold difference 2460, illustrated in FIG.20B. Accordingly, in this example, at or near (e.g., within a brief timeinterval of a few seconds or minutes following t₁), the criteria aresatisfied and the descriptor 2212 associated with descriptor values2452, 2454 can generally be used to re-identify the first and thirdpeople 2402, 2406.

When the criteria are satisfied for distinguishing the first person 2402from the third person 2406 based on the first descriptor 2212 (as is thecase at t₁), the descriptor comparator 2214 may compare the firstdescriptor 2212 for the first person 2402 to each of the correspondingpredetermined descriptors 2210 (i.e., for all identifiers 2208).However, in some embodiments, comparator 2214 may compare the firstdescriptor 2212 for the first person 2402 to predetermined descriptors2210 for only a select subset of the identifiers 2208. The subset may beselected using the candidate list 2206 for the person that is beingre-identified (see, e.g., step 2608 of method 2600 described below withrespect to FIG. 21). For example, the person's candidate list 2206 mayindicate that only a subset (e.g., two, three, or so) of a larger numberof identifiers 2208 are likely to be associated with the tracked objectposition 11202 that requires re-identification. Based on thiscomparison, the tracking subsystem 2200 may identify the predetermineddescriptor 2210 that is most similar to the first descriptor 2212. Forexample, the tracking subsystem 2200 may determine that a firstidentifier 2208 corresponds to the first person 2402 by, for each memberof the set (or the determined subset) of the predetermined descriptors2210, calculating an absolute value of a difference in a value of thefirst descriptor 2212 and a value of the predetermined descriptor 2210.The first identifier 2208 may be selected as the identifier 2208associated with the smallest absolute value.

Referring again to FIG. 20A, at time t₂, a second collision event occursat position 2418 between people 2402, 2406. Turning back to FIG. 20B,the descriptor values 2452, 2454 have a relatively small difference 2458at time t₂ (e.g., compared to difference 2456 at time t₁), which is lessthan the threshold value 2460. Thus, at time t₂, the descriptor 2212associated with descriptor values 2452, 2454 generally cannot be used tore-identify the first and third people 2402, 2406, and the criteria forusing the first descriptor 2212 are not satisfied. Instead, a different,and likely a “higher-cost” descriptor 2212 (e.g., a model-baseddescriptor 2324) should be used to re-identify the first and thirdpeople 2402, 2406 at time t₂.

For example, when the criteria are not satisfied for distinguishing thefirst person 2402 from the third person 2406 based on the firstdescriptor 2212 (as is the case in this example at time t₂), thetracking subsystem 2200 determines a new descriptor 2212 for the firstperson 2402. The new descriptor 2212 is typically a value or vectorgenerated by an artificial neural network configured to identify peoplein top-view images (e.g., a model-based descriptor 2324 of FIG. 19). Thetracking subsystem 2200 may determine, based on the new descriptor 2212,that a first identifier 2208 from the predetermined identifiers 2208 (ora subset determined based on the candidate list 2206, as describedabove) corresponds to the first person 2402. For example, the trackingsubsystem 2200 may determine that the first identifier 2208 correspondsto the first person 2402 by, for each member of the set (or subset) ofpredetermined identifiers 2208, calculating an absolute value of adifference in a value of the first identifier 2208 and a value of thepredetermined descriptors 2210. The first identifier 2208 may beselected as the identifier 2208 associated with the smallest absolutevalue.

In cases where the second descriptor 2212 cannot be used to reliablyre-identify the first person 2402 using the approach described above,the tracking subsystem 2200 may determine a measured descriptor 2212 forall of the “candidate identifiers” of the first person 2402. Thecandidate identifiers generally refer to the identifiers 2208 of people(e.g., or other tracked objects) that are known to be associated withidentifiers 2208 appearing in the candidate list 2206 of the firstperson 2402 (e.g., as described above with respect to FIGS. 16 and 17).For instance, the candidate identifiers may be identifiers 2208 oftracked people (i.e., at tracked object positions 11202) that appear inthe candidate list 2206 of the person being re-identified. FIG. 20Cillustrates how predetermined descriptors 2462, 2464, 2466 for a first,second, and third identifier 2208 may be compared to each of themeasured descriptors 2468, 2470, 2472 for people 2402, 2404, 2406. Thecomparison may involve calculating a cosine similarity value between avectors associated with the descriptors. Based on the results of thecomparison, each person 2402, 2404, 2406 is assigned the identifier 2208corresponding to the best-matching predetermined descriptor 2462, 2464,2466. A best matching descriptor may correspond to a highest cosinesimilarity value (i.e., nearest to one).

FIG. 21 illustrates a method 2600 for re-identifying tracked peopleusing tracking subsystem 2200 illustrated in FIG. 18 and describedabove. The method 2600 may begin at step 2602 where the trackingsubsystem 2200 receives top-view image frames from one or more cameras305. At step 2604, the tracking subsystem 2200 tracks a first person2402 and one or more other people (e.g., people 2404, 2406) in the store100 using at least a portion of the top-view images generated by thecameras 305. For instance, tracking may be performed as described abovewith respect to FIGS. 13-15, or using any appropriate object trackingalgorithm. The tracking subsystem 2200 may periodically determineupdated predetermined descriptors associated with the identifiers 2208(e.g., as described with respect to update 2218 of FIG. 18). In someembodiments, the tracking subsystem 2200, in response to determining theupdated descriptors, determines that one or more of the updatedpredetermined descriptors is different by at least a threshold amountfrom a corresponding previously predetermined descriptor 2210. In thiscase, the tracking subsystem 2200 may save both the updated descriptorand the corresponding previously predetermined descriptor 2210. This mayallow for improved re-identification when characteristics of the peoplebeing tracked may change intermittently during tracking.

At step 2606, the tracking subsystem 2200 determines whetherre-identification of the first tracked person 2402 is needed. This maybe based on a determination that contours have merged in an image frame(e.g., as illustrated by merged contour 1120 of FIG. 11) or on adetermination that a first person 2402 and a second person 2404 arewithin a threshold distance (e.g., distance 2218 b of FIG. 18) of eachother, as described above. In some embodiments, a candidate list 2206may be used to determine that re-identification of the first person 2402is required. For instance, if a highest probability from the candidatelist 2206 associated with the tracked person 2402 is less than athreshold value (e.g., 70%), re-identification may be needed (see alsoFIGS. 16-17 and the corresponding description above). Ifre-identification is not needed, the tracking subsystem 2200 generallycontinues to track people in the space (e.g., by returning to step2604).

If the tracking subsystem 2200 determines at step 2606 thatre-identification of the first tracked person 2402 is needed, thetracking subsystem 2200 may determine candidate identifiers for thefirst tracked person 2402 at step 2608. The candidate identifiersgenerally include a subset of all of the identifiers 2208 associatedwith tracked people in the store 100, and the candidate identifiers maybe determined based on the candidate list 2206 for the first trackedperson 2402. In other words, the candidate 2206 which are most likely toinclude the correct identifier 2208 for the first tracked person 2402based on a history of movements of the first tracked person 2402 andinteractions of the first tracked person 2402 with the one or more othertracked people 2404, 2406 in the store 100 (e.g., based on the candidatelist 2206 that is updated in response to these movements andinteractions).

At step 2610, the tracking subsystem 2200 determines a first descriptor2212 for the first tracked person 2402. For example, the trackingsubsystem 2200 may receive, from a first camera 305, a first top-viewimage of the first person 2402 (e.g., such as image 2302 of FIG. 19).For instance, as illustrated in the example of FIG. 19, in someembodiments, the image 2302 used to determine the descriptor 2212includes the representation 2304 a of the object within aregion-of-interest 2306 within the full frame of the image 2302. Thismay provide for more reliable descriptor 2212 determination. In someembodiments, the image data 2204 include depth data (i.e., image data atdifferent depths). In such embodiments, the tracking subsystem 2200 maydetermine the descriptor 2212 based on a depth region-of-interest, wherethe depth region-of-interest corresponds to depths in the imageassociated with the head of person 2402. In these embodiments,descriptors 2212 may be determined that are associated withcharacteristics or features of the head of the person 2402.

At step 2612, the tracking subsystem 2200 may determine whether thefirst descriptor 2212 can be used to distinguish the first person 2402from the candidate identifiers (e.g., one or both of people 2404, 2406)by, for example, determining whether certain criteria are satisfied fordistinguishing the first person 2402 from the candidates based on thefirst descriptor 2212. In some embodiments, the criteria are notsatisfied when a difference, determined during a time intervalassociated with the collision event, between the first descriptor 2212and corresponding descriptors 2210 of the candidates is less than aminimum value, as described in greater detail above with respect toFIGS. 20A,B.

If the first descriptor can be used to distinguish the first person 2402from the candidates (e.g., as was the case at time t₁ in the example ofFIG. 20A,B), the method 2600 proceeds to step 2614 at which point thetracking subsystem 2200 determines an updated identifier for the firstperson 2402 based on the first descriptor 2212. For example, thetracking subsystem 2200 may compare (e.g., using comparator 2214) thefirst descriptor 2212 to the set of predetermined descriptors 2210 thatare associated with the candidate objects determined for the firstperson 2402 at step 2608. In some embodiments, the first descriptor 2212is a data vector associated with characteristics of the first person inthe image (e.g., a vector determined using a texture operator such asthe LBPH algorithm), and each of the predetermined descriptors 2210includes a corresponding predetermined data vector (e.g., determined foreach tracked person 2402, 2404, 2406 upon entering the store 100). Insuch embodiments, the tracking subsystem 2200 compares the firstdescriptor 2212 to each of the predetermined descriptors 2210 associatedwith the candidate objects by calculating a cosine similarity valuebetween the first data vector and each of the predetermined datavectors. The tracking subsystem 2200 determines the updated identifieras the identifier 2208 of the candidate object with the cosinesimilarity value nearest one (i.e., the vector that is most “similar” tothe vector of the first descriptor 2212).

At step 2616, the identifiers 2208 of the other tracked people 2404,2406 may be updated as appropriate by updating other people's candidatelists 2206. For example, if the first tracked person 2402 was found tobe associated with an identifier 2208 that was previously associatedwith the second tracked person 2404. Steps 2608 to 2614 may be repeatedfor the second person 2404 to determine the correct identifier 2208 forthe second person 2404. In some embodiments, when the identifier 2208for the first person 2402 is updated, the identifiers 2208 for people(e.g., one or both of people 2404 and 2406) that are associated with thefirst person's candidate list 2206 are also updated at step 2616. As anexample, the candidate list 2206 of the first person 2402 may have anon-zero probability that the first person 2402 is associated with asecond identifier 2208 originally linked to the second person 2404 and athird probability that the first person 2402 is associated with a thirdidentifier 2208 originally linked to the third person 2406. In thiscase, after the identifier 2208 of the first person 2402 is updated, theidentifiers 2208 of the second and third people 2404, 2406 may also beupdated according to steps 2608-2614.

If, at step 2612, the first descriptor 2212 cannot be used todistinguish the first person 2402 from the candidates (e.g., as was thecase at time t₂ in the example of FIG. 20A,B), the method 2600 proceedsto step 2618 to determine a second descriptor 2212 for the first person2402. As described above, the second descriptor 2212 may be a“higher-level” descriptor such as a model-based descriptor 2324 of FIG.19). For example, the second descriptor 2212 may be less efficient(e.g., in terms of processing resources required) to determine than thefirst descriptor 2212. However, the second descriptor 2212 may be moreeffective and reliable, in some cases, for distinguishing betweentracked people.

At step 2620, the tracking system 2200 determines whether the seconddescriptor 2212 can be used to distinguish the first person 2402 fromthe candidates (from step 2618) using the same or a similar approach tothat described above with respect to step 2612. For example, thetracking subsystem 2200 may determine if the cosine similarity valuesbetween the second descriptor 2212 and the predetermined descriptors2210 are greater than a threshold cosine similarity value (e.g., of0.5). If the cosine similarity value is greater than the threshold, thesecond descriptor 2212 generally can be used.

If the second descriptor 2212 can be used to distinguish the firstperson 2402 from the candidates, the tracking subsystem 2200 proceeds tostep 2622, and the tracking subsystem 2200 determines the identifier2208 for the first person 2402 based on the second descriptor 2212 andupdates the candidate list 2206 for the first person 2402 accordingly.The identifier 2208 for the first person 2402 may be determined asdescribed above with respect to step 2614 (e.g., by calculating a cosinesimilarity value between a vector corresponding to the first descriptor2212 and previously determined vectors associated with the predetermineddescriptors 2210). The tracking subsystem 2200 then proceeds to step2616 described above to update identifiers 2208 (i.e., via candidatelists 2206) of other tracked people 2404, 2406 as appropriate.

Otherwise, if the second descriptor 2212 cannot be used to distinguishthe first person 2402 from the candidates, the tracking subsystem 2200proceeds to step 2624, and the tracking subsystem 2200 determines adescriptor 2212 for all of the first person 2402 and all of thecandidates. In other words, a measured descriptor 2212 is determined forall people associated with the identifiers 2208 appearing in thecandidate list 2206 of the first person 2402 (e.g., as described abovewith respect to FIG. 20C). At step 2626, the tracking subsystem 2200compares the second descriptor 2212 to predetermined descriptors 2210associated with all people related to the candidate list 2206 of thefirst person 2402. For instance, the tracking subsystem 2200 maydetermine a second cosine similarity value between a second data vectordetermined using an artificial neural network and each correspondingvector from the predetermined descriptor values 2210 for the candidates(e.g., as illustrated in FIG. 20C, described above). The trackingsubsystem 2200 then proceeds to step 2628 to determine and update theidentifiers 2208 of all candidates based on the comparison at step 2626before continuing to track people 2402, 2404, 2406 in the store 100(e.g., by returning to step 2604).

Modifications, additions, or omissions may be made to method 2600depicted in FIG. 21. Method 2600 may include more, fewer, or othersteps. For example, steps may be performed in parallel or in anysuitable order. While at times discussed as tracking system 2200 (e.g.,by camera server 225 and/or camera client(s) 220) or components thereofperforming steps, any suitable system or components of the system mayperform one or more steps of the method 2600.

While the preceding examples and explanations are described with respectto particular use cases within a retail environment, one of ordinaryskill in the art would readily appreciate that the previously describedconfigurations and techniques may also be applied to other applicationsand environments. Examples of other applications and environmentsinclude, but are not limited to, security applications, surveillanceapplications, object tracking applications, people trackingapplications, occupancy detection applications, logistics applications,warehouse management applications, operations research applications,product loading applications, retail applications, roboticsapplications, computer vision applications, manufacturing applications,safety applications, quality control applications, food distributingapplications, retail product tracking applications, mappingapplications, simultaneous localization and mapping (SLAM) applications,3D scanning applications, autonomous vehicle applications, virtualreality applications, augmented reality applications, or any othersuitable type of application.

Although the present disclosure includes several embodiments, a myriadof changes, variations, alterations, transformations, and modificationsmay be suggested to one skilled in the art, and it is intended that thepresent disclosure encompass such changes, variations, alterations,transformations, and modifications as fall within the scope of theappended claims.

While several embodiments have been provided in the present disclosure,it should be understood that the disclosed systems and methods might beembodied in many other specific forms without departing from the spiritor scope of the present disclosure. The present examples are to beconsidered as illustrative and not restrictive, and the intention is notto be limited to the details given herein. For example, the variouselements or components may be combined or integrated in another systemor certain features may be omitted, or not implemented.

In addition, techniques, systems, subsystems, and methods described andillustrated in the various embodiments as discrete or separate may becombined or integrated with other systems, modules, techniques, ormethods without departing from the scope of the present disclosure.Other items shown or discussed as coupled or directly coupled orcommunicating with each other may be indirectly coupled or communicatingthrough some interface, device, or intermediate component whetherelectrically, mechanically, or otherwise. Other examples of changes,substitutions, and alterations are ascertainable by one skilled in theart and could be made without departing from the spirit and scopedisclosed herein.

To aid the Patent Office, and any readers of any patent issued on thisapplication in interpreting the claims appended hereto, applicants notethat they do not intend any of the appended claims to invoke 35 U.S.C. §112(f) as it exists on the date of filing hereof unless the words “meansfor” or “step for” are explicitly used in the particular claim.

What is claimed is:
 1. A system comprising: an array of cameraspositioned above a space, wherein: each camera of the array of camerasis operatively coupled with a camera client from an array of cameraclients; and each camera of the array of cameras is configured tocapture a video of a portion of the space, the space containing aperson; the array of camera clients operably coupled with the array ofcameras; wherein: a first camera client of the array of camera clientsis operably coupled with a first camera and configured to: receive afirst plurality of frames of a first video from the first camera,wherein at least one frame of the first plurality of frames shows theperson within the space; generate a timestamp when each frame from thefirst plurality of frames is received by the first camera client; sendthe first plurality of frames labeled with one or more correspondingtimestamps and an identifier number of the first camera client to afirst server from among a plurality of cluster servers; generate a firstplurality of tracks by performing a local position tracking of theperson, wherein at least one track of the first plurality of tracksindicates a location of the person in the least one frame of the firstplurality of frames; and send the first plurality of tracks labeled withone or more corresponding timestamps and the identifier number of thefirst camera to a second server from among the plurality of clusterservers.
 2. The system of claim 1, wherein: a second camera client ofthe array of camera clients is operably coupled with a second camera andseparate from the first camera client, the second camera clientconfigured to: receive a second plurality of frames of a second videofrom the second camera, wherein at least one frame of the secondplurality of frames shows the person within the space; generate atimestamp when each frame of the second plurality of frames is receivedby the second camera client; send the second plurality of frames labeledwith one or more corresponding timestamps and an identifier number ofthe second camera to the first server; generate a second plurality oftracks by performing a local position tracking of the person; wherein atleast one track of the second plurality of tracks indicates a locationof the person in the least one frame of the second plurality of frames;and send the second plurality of tracks labeled with one or morecorresponding timestamps and an identifier number of the second camerato the second server.
 3. The system of claim 2, further comprising: thefirst server, operably coupled with at least one of the first cameraclient and the second camera client, the first server configured to:receive at least one frame from the first and second plurality offrames; and store the at least one frame such that the at least oneframe is retrievable; the second server, operably coupled with at leastone of the first camera client and the second camera client, the secondserver configured to: receive at least one track from the first andsecond plurality of tracks; and store the at least one track such the atleast one track is retrievable.
 4. The system of claim 3, wherein aparticular frame from the at least one frame is retrievable using one ormore corresponding labels comprising an identifier number of a cameraassociated with the particular frame and a timestamp associated with theparticular frame.
 5. The system of claim 3, wherein a particular trackfrom the at least one track is retrievable using one or morecorresponding labels comprising an identifier number of a cameraassociated with the particular track, a timestamp associated with theparticular track, a particular historical detection associated with theperson detected in the particular track, and a particular trackingidentification detected in the particular track.
 6. The system of claim1, wherein: the first plurality of frames comprises a first plurality ofcolor frames and a first plurality of depth frames; the first pluralityof color frames corresponds to visual colors of objects in the space;and the first plurality of depth frames corresponds to distances ofobjects in the space from the first camera.
 7. The system of claim 6,wherein generating the first plurality of tracks comprises performingthe local position tracking of the person in the first plurality ofdepth frames, and wherein: for a first depth frame of the firstplurality of depth frames, generating a first track of the firstplurality of tracks comprises: detecting a first contour associated withthe person; determining, based on pixel coordinates of the firstcontour, a first bounding area around the person shown in the firstdepth frame; determining, based on the first bounding area, firstcoordinates of the person in the first depth frame; and associating afirst tracking identification to the person, wherein: the first trackingidentification is linked to historical detections associated with theperson, and the historical detections associated with the personcomprise at least one of a contour, a bounding area, and a segmentationmask associated with the person; for a second depth frame of the firstplurality of depth frames, generating a second track of the firstplurality of tracks comprises: detecting a second contour associatedwith the person; determining, based on pixel coordinates of the secondcontour, a second bounding area around the person shown in the seconddepth frame; determining, based on the second bounding area, secondcoordinates of the person in the second depth frame; determining whetherthe second bounding area corresponds to the first bounding area; and inresponse to determining that the second bounding area corresponds to thefirst bounding area, associating the first tracking identification tothe person.
 8. The system of claim 7, wherein at least one track fromthe first plurality of tracks is further labeled with the historicaldetections associated with the person and the first trackingidentification associated with the person.
 9. A method comprising:receiving, by a first camera client, a first plurality of frames of afirst video from a first camera, wherein at least one frame of the firstplurality of frames shows the person within a space; generating, by thefirst camera client, a timestamp when each frame from the firstplurality of frames is received from the first camera; sending, by thefirst camera client, the first plurality of frames labeled with one ormore corresponding timestamps and an identifier number of the firstcamera client to a first server from among a plurality of clusterservers; generating, by the first camera client, a first plurality oftracks by performing a local position tracking of the person, wherein atleast one track of the first plurality of tracks indicates a location ofthe person in the at least one frame of the first plurality of frames;and sending, by the first camera client, the first plurality of trackslabeled with one or more corresponding timestamps and the identifiernumber of the first camera to a second server from among the pluralityof cluster servers.
 10. The method of claim 9, further comprising:receiving, by a second camera client, a second plurality of frames of asecond video from a second camera, wherein at least one frame of thesecond plurality of frames shows the person within the space;generating, by the second camera client, a timestamp when each framefrom the second plurality of frames is received from the second camera;sending, by the second camera client, the second plurality of frameslabeled with one or more corresponding timestamps and an identifiernumber of the second camera to the first server; generating, by thesecond camera client, a second plurality of tracks by performing a localposition tracking of the person; wherein at least one track of thesecond plurality of tracks indicates a location of the person in the atleast one frame of the second plurality of frames; and sending, by thesecond camera client, the first plurality of tracks labeled with one ormore corresponding timestamps and the identifier number of the secondcamera to the second server.
 11. The method of claim 10, furthercomprising: receiving, by the first server, at least one frame from thefirst and second plurality of frames; storing, by the first server, theat least one frame such that the at least one frame is retrievable;receiving, by the second server, at least one track from the first andsecond plurality of tracks; and storing, by the second server, the atleast one track such that the at least one track is retrievable.
 12. Themethod of claim 11, wherein a particular frame from the at least oneframe is retrievable using one or more corresponding labels comprisingan identifier number of a camera associated with the particular frameand a timestamp associated with the particular frame.
 13. The method ofclaim 11, wherein a particular track from the at least one track isretrievable using one or more corresponding labels comprising anidentifier number of a camera associated with the particular track, atimestamp associated with the particular track, a particular historicaldetection associated with the person detected in the particular track,and a particular tracking identification detected in the particulartrack.
 14. The method of claim 9, wherein: the first plurality of framescomprises a first plurality of color frames and a first plurality ofdepth frames; the first plurality of color frames corresponds to visualcolors of objects in the space; and the first plurality of depth framescorresponds to distances of objects in the space from the first camera.15. The method of claim 14, wherein generating the first plurality oftracks comprises performing the local position tracking of the person inthe first plurality of depth frames, wherein: for a first depth frame ofthe first plurality of depth frames, generating a first track of thefirst plurality of tracks comprises: detecting a first contourassociated with the person; determining, based on pixel coordinates ofthe first contour, a first bounding area around the person shown in thefirst depth frame; determining, based on the first bounding area, firstcoordinates of the person in the first depth frame; and associating afirst tracking identification to the person, wherein: the first trackingidentification is linked to historical detections associated with theperson, and the historical detections associated with the personcomprise at least one of a contour, a bounding area, and a segmentationmask associated with the person; for a second depth frame of the firstplurality of depth frames, generating a second track of the firstplurality of tracks comprises: detecting a second contour associatedwith the person; determining, based on pixel coordinates of the secondcontour, a second bounding area around the person shown in the seconddepth frame; determining, based on the second bounding area, secondcoordinates of the person in the second depth frame; determining whetherthe second bounding area corresponds to the first bounding area; and inresponse to determining that the second bounding area corresponds to thefirst bounding area, associating the first tracking identification tothe person.
 16. The method of claim 15, wherein at least one track fromthe first plurality of tracks is further labeled with the historicaldetections associated with the person and the first trackingidentification associated with the person.
 17. A computer programcomprising executable instructions stored in a non-transitorycomputer-readable medium that when executed by a processor causes theprocessor to: receive, by a first camera client, a first plurality offrames of a first video from a first camera, wherein at least one frameof the first plurality of frames shows the person within a space;generate, by the first camera client, a timestamp when each frame fromthe first plurality of frames is received from the first camera; send,by the first camera client, the first plurality of frames labeled withone or more corresponding timestamps and an identifier number of thefirst camera client to a first server from among a plurality of clusterservers; generate, by the first camera client, a first plurality oftracks by performing a local position tracking of the person, wherein atleast one track of the first plurality of tracks indicates a location ofthe person in the at least one frame of the first plurality of frames;and send, by the first camera client, the first plurality of trackslabeled with one or more corresponding timestamps and the identifiernumber of the first camera to a second server from among the pluralityof cluster servers.
 18. The computer program of claim 17, wherein theinstructions when executed by the processor, further cause the processorto: receive, by a second camera client, a second plurality of frames ofa second video from a second camera, wherein at least one frame of thesecond plurality of frames shows the person within the space; generate,by the second camera client, a timestamp when each frame from the secondplurality of frames is received from the second camera; send, by thesecond camera client, the second plurality of frames labeled with one ormore corresponding timestamps and an identifier number of the secondcamera to the first server; generate, by the second camera client, asecond plurality of tracks by performing a local position tracking ofthe person; wherein at least one track of the second plurality of tracksindicates a location of the person in the at least one frame of thesecond plurality of frames; and send, by the second camera client, thefirst plurality of tracks labeled with one or more correspondingtimestamps and the identifier number of the second camera to the secondserver.
 19. The computer program of claim 18, wherein the instructionswhen executed by the processor, further cause the processor to: receive,by the first server, at least one frame from the first and secondplurality of frames; store, by the first server, the at least one framesuch that the at least one frame is retrievable; receive, by the secondserver, at least one track from the first and second plurality oftracks; and store, by the second server, the at least one track suchthat the at least one track is retrievable.
 20. The computer program ofclaim 19, wherein: a particular frame from the at least one frame isretrievable using one or more corresponding labels comprising anidentifier number of a camera associated with the particular frame and atimestamp associated with the particular frame, and a particular trackfrom the at least one track is retrievable using one or morecorresponding labels comprising an identifier number of a cameraassociated with the particular track, a timestamp associated with theparticular track, a particular historical detection associated with theperson detected in the particular track, and a particular trackingidentification detected in the particular track.